==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-db4967764-9zt9t Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-1b0769c8-7vmj/10.142.0.111 Start Time: Wed, 17 Apr 2024 19:19:42 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=db4967764 skaffold.dev/run-id=f770c1a5-e07c-4b72-be34-e34a0aea1817 Annotations: Status: Running IP: 10.106.42.183 IPs: IP: 10.106.42.183 Controlled By: ReplicaSet/lodemon-db4967764 Containers: lodemon: Container ID: containerd://2679007b75fffe9851a742901f4cd6c16870a28ab20fbba587b1fbe70b0e0e55 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:2febaeab396557db3e0fc07c0501fa786a35d3473828be1a84a3d1610beec5de Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Wed, 17 Apr 2024 19:19:43 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n9dkm (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-n9dkm: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 41m default-scheduler Successfully assigned xlou/lodemon-db4967764-9zt9t to gke-xlou-cdm-default-pool-1b0769c8-7vmj Normal Pulling 41m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 41m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 243.042609ms (243.048442ms including waiting) Normal Created 41m kubelet Created container lodemon Normal Started 41m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.42.183:8080 Press CTRL+C to quit 20:20:50 INFO 20:20:50 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:50 INFO [loop_until]: OK (rc = 0) 20:20:50 DEBUG --- stdout --- 20:20:50 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:50 DEBUG --- stderr --- 20:20:50 DEBUG 20:20:50 INFO 20:20:50 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:50 INFO [loop_until]: OK (rc = 0) 20:20:50 DEBUG --- stdout --- 20:20:50 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:50 DEBUG --- stderr --- 20:20:50 DEBUG 20:20:50 INFO 20:20:50 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:50 INFO [loop_until]: OK (rc = 0) 20:20:50 DEBUG --- stdout --- 20:20:50 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:50 DEBUG --- stderr --- 20:20:50 DEBUG 20:20:50 INFO 20:20:50 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:50 INFO [loop_until]: OK (rc = 0) 20:20:50 DEBUG --- stdout --- 20:20:50 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:50 DEBUG --- stderr --- 20:20:50 DEBUG 20:20:50 INFO 20:20:50 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:50 INFO [loop_until]: OK (rc = 0) 20:20:50 DEBUG --- stdout --- 20:20:50 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:50 DEBUG --- stderr --- 20:20:50 DEBUG 20:20:50 INFO 20:20:50 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:50 INFO [loop_until]: OK (rc = 0) 20:20:50 DEBUG --- stdout --- 20:20:50 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:50 DEBUG --- stderr --- 20:20:50 DEBUG 20:20:50 INFO 20:20:50 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:50 INFO [loop_until]: OK (rc = 0) 20:20:50 DEBUG --- stdout --- 20:20:50 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:50 DEBUG --- stderr --- 20:20:50 DEBUG 20:20:51 INFO 20:20:51 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:51 INFO [loop_until]: OK (rc = 0) 20:20:51 DEBUG --- stdout --- 20:20:51 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:51 DEBUG --- stderr --- 20:20:51 DEBUG 20:20:51 INFO 20:20:51 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:51 INFO [loop_until]: OK (rc = 0) 20:20:51 DEBUG --- stdout --- 20:20:51 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:51 DEBUG --- stderr --- 20:20:51 DEBUG 20:20:51 INFO 20:20:51 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:51 INFO [loop_until]: OK (rc = 0) 20:20:51 DEBUG --- stdout --- 20:20:51 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:51 DEBUG --- stderr --- 20:20:51 DEBUG 20:20:51 INFO 20:20:51 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:51 INFO [loop_until]: OK (rc = 0) 20:20:51 DEBUG --- stdout --- 20:20:51 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:51 DEBUG --- stderr --- 20:20:51 DEBUG 20:20:51 INFO 20:20:51 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:51 INFO [loop_until]: OK (rc = 0) 20:20:51 DEBUG --- stdout --- 20:20:51 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:51 DEBUG --- stderr --- 20:20:51 DEBUG 20:20:51 INFO 20:20:51 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:51 INFO [loop_until]: OK (rc = 0) 20:20:51 DEBUG --- stdout --- 20:20:51 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:51 DEBUG --- stderr --- 20:20:51 DEBUG 20:20:51 INFO 20:20:51 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:51 INFO [loop_until]: OK (rc = 0) 20:20:51 DEBUG --- stdout --- 20:20:51 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:51 DEBUG --- stderr --- 20:20:51 DEBUG 20:20:51 INFO 20:20:51 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:52 INFO [loop_until]: OK (rc = 0) 20:20:52 DEBUG --- stdout --- 20:20:52 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:52 DEBUG --- stderr --- 20:20:52 DEBUG 20:20:52 INFO 20:20:52 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:52 INFO [loop_until]: OK (rc = 0) 20:20:52 DEBUG --- stdout --- 20:20:52 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:52 DEBUG --- stderr --- 20:20:52 DEBUG 20:20:52 INFO 20:20:52 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:52 INFO [loop_until]: OK (rc = 0) 20:20:52 DEBUG --- stdout --- 20:20:52 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:52 DEBUG --- stderr --- 20:20:52 DEBUG 20:20:52 INFO 20:20:52 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:52 INFO [loop_until]: OK (rc = 0) 20:20:52 DEBUG --- stdout --- 20:20:52 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:52 DEBUG --- stderr --- 20:20:52 DEBUG 20:20:52 INFO 20:20:52 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:52 INFO [loop_until]: OK (rc = 0) 20:20:52 DEBUG --- stdout --- 20:20:52 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:52 DEBUG --- stderr --- 20:20:52 DEBUG 20:20:52 INFO 20:20:52 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:52 INFO [loop_until]: OK (rc = 0) 20:20:52 DEBUG --- stdout --- 20:20:52 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:52 DEBUG --- stderr --- 20:20:52 DEBUG 20:20:52 INFO 20:20:52 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:52 INFO [loop_until]: OK (rc = 0) 20:20:52 DEBUG --- stdout --- 20:20:52 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:52 DEBUG --- stderr --- 20:20:52 DEBUG 20:20:52 INFO 20:20:52 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:53 INFO [loop_until]: OK (rc = 0) 20:20:53 DEBUG --- stdout --- 20:20:53 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:53 DEBUG --- stderr --- 20:20:53 DEBUG 20:20:53 INFO 20:20:53 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:53 INFO [loop_until]: OK (rc = 0) 20:20:53 DEBUG --- stdout --- 20:20:53 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:53 DEBUG --- stderr --- 20:20:53 DEBUG 20:20:53 INFO 20:20:53 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:53 INFO [loop_until]: OK (rc = 0) 20:20:53 DEBUG --- stdout --- 20:20:53 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:53 DEBUG --- stderr --- 20:20:53 DEBUG 20:20:53 INFO 20:20:53 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:53 INFO [loop_until]: OK (rc = 0) 20:20:53 DEBUG --- stdout --- 20:20:53 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:53 DEBUG --- stderr --- 20:20:53 DEBUG 20:20:53 INFO 20:20:53 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:53 INFO [loop_until]: OK (rc = 0) 20:20:53 DEBUG --- stdout --- 20:20:53 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:53 DEBUG --- stderr --- 20:20:53 DEBUG 20:20:53 INFO 20:20:53 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:20:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:53 INFO [loop_until]: OK (rc = 0) 20:20:53 DEBUG --- stdout --- 20:20:53 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:20:53 DEBUG --- stderr --- 20:20:53 DEBUG 20:20:53 INFO Initializing monitoring instance threads 20:20:53 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 20:20:53 INFO Starting instance threads 20:20:53 INFO 20:20:53 INFO Thread started 20:20:53 INFO [loop_until]: kubectl --namespace=xlou top node 20:20:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:53 INFO 20:20:53 INFO Thread started 20:20:53 INFO [loop_until]: kubectl --namespace=xlou top pods 20:20:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 DEBUG --- status code --- 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 INFO Thread started 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG --- status code --- 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 DEBUG --- status code --- 20:20:53 INFO Thread started 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 INFO Thread started 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG --- status code --- 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 DEBUG --- status code --- 20:20:53 INFO Thread started 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 DEBUG { 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:20:53 INFO Thread started 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:20:53 DEBUG ----- output ----- 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG ----- output ----- 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG ----- output ----- 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG { 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG { 20:20:53 DEBUG { 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:20:53 DEBUG "data": { 20:20:53 DEBUG { 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 DEBUG { 20:20:53 INFO Thread started 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG { 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG "status": "success", 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 DEBUG "data": { 20:20:53 INFO Thread started 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG "data": { 20:20:53 DEBUG "data": { 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:20:53 DEBUG { 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG { 20:20:53 DEBUG "result": [ 20:20:53 DEBUG "data": { 20:20:53 INFO [loop_until]: OK (rc = 0) 20:20:53 DEBUG "data": { 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG "resultType": "vector", 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG "data": { 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG ----- output ----- 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG { 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG { 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG --- stdout --- 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG "result": [ 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG "result": [ 20:20:53 DEBUG "result": [ 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG { 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG "data": { 20:20:53 DEBUG "metric": { 20:20:53 DEBUG "result": [ 20:20:53 DEBUG "result": [ 20:20:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 8m 954Mi am-b59c765b7-bgsxp 9m 2929Mi am-b59c765b7-ch5q4 8m 2610Mi ds-cts-0 8m 7516Mi ds-cts-1 5m 7383Mi ds-cts-2 5m 7660Mi ds-idrepo-0 11m 2985Mi ds-idrepo-1 7m 1538Mi ds-idrepo-2 18m 1516Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 8m 965Mi idm-b85f9d978-wnhxb 5m 1009Mi lodemon-db4967764-9zt9t 1m 54Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1m 290Mi 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 DEBUG { 20:20:53 INFO Thread started 20:20:53 DEBUG { 20:20:53 DEBUG { 20:20:53 INFO [loop_until]: OK (rc = 0) 20:20:53 DEBUG "result": [ 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG { 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG "data": { 20:20:53 DEBUG "data": { 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG "data": { 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:20:53 DEBUG { 20:20:53 DEBUG { 20:20:53 DEBUG --- stderr --- 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG { 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG "status": "success", 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 DEBUG "metric": { 20:20:53 INFO Thread started 20:20:53 DEBUG "metric": { 20:20:53 DEBUG --- stdout --- 20:20:53 DEBUG { 20:20:53 DEBUG { 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG "result": [ 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG "result": [ 20:20:53 DEBUG }, 20:20:53 DEBUG "metric": { 20:20:53 DEBUG "metric": { 20:20:53 DEBUG 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG --- http response --- 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG "metric": { 20:20:53 DEBUG "data": { 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:20:53 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:20:53 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713381653" 20:20:53 INFO Thread started 20:20:53 DEBUG "metric": { 20:20:53 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 5519Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 5832Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 5140Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 2985Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 63m 0% 9652Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 164m 0% 4996Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 71m 0% 3577Mi 3% gke-xlou-cdm-ds-0f601751-cf7b 64m 0% 2868Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 4751Mi 4% gke-xlou-cdm-ds-0f601751-j488 78m 0% 4012Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 10860Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 94m 0% 4570Mi 4% gke-xlou-cdm-ds-0f601751-nmxm 65m 0% 9255Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 72m 0% 4105Mi 7% 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG "data": { 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:20:53 DEBUG "result": [ 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG { 20:20:53 DEBUG "result": [ 20:20:53 DEBUG { 20:20:53 DEBUG "value": [ 20:20:53 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:20:53 DEBUG "instance": "10.142.0.107:9100" 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG { 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG }, 20:20:53 DEBUG }, 20:20:53 DEBUG "instance": "10.142.0.107:9100" 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381653" 20:20:53 DEBUG --- stderr --- 20:20:53 INFO Thread started 20:20:53 DEBUG "data": { 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG { 20:20:53 DEBUG "metric": { 20:20:53 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "am-b59c765b7-bgsxp" }, "value": [ 1713381653, "0" ] } ] } } 20:20:53 DEBUG { 20:20:53 DEBUG "metric": { 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG 20:20:53 DEBUG }, 20:20:53 DEBUG }, 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG }, 20:20:53 DEBUG "result": [ 20:20:53 DEBUG { 20:20:53 DEBUG "value": [ 20:20:53 DEBUG "value": [ 20:20:53 DEBUG }, 20:20:53 DEBUG 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381653" 20:20:53 DEBUG "resultType": "vector", 20:20:53 INFO Thread started 20:20:53 DEBUG "result": [ 20:20:53 DEBUG { 20:20:53 DEBUG "metric": { 20:20:53 DEBUG "pod": "ds-cts-0" 20:20:53 DEBUG "metric": { 20:20:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG 20:20:53 DEBUG "value": [ 20:20:53 DEBUG "value": [ 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG "data": { 20:20:53 DEBUG { 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG "value": [ 20:20:53 DEBUG { 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG "value": [ 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG "result": [ 20:20:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381653" 20:20:53 DEBUG { 20:20:53 INFO Thread started 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG "instance": "10.142.0.107:9100" 20:20:53 DEBUG }, 20:20:53 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:20:53 DEBUG }, 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG "result": [ 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:20:53 DEBUG "metric": { 20:20:53 DEBUG "data": { 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG { 20:20:53 DEBUG "metric": { 20:20:53 INFO All threads has been started 127.0.0.1 - - [17/Apr/2024 20:20:53] "GET /monitoring/start HTTP/1.1" 200 - 20:20:53 DEBUG "data": { 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG }, 20:20:53 DEBUG "value": [ 20:20:53 DEBUG }, 20:20:53 DEBUG "value": [ 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG [...] 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:20:53 ERROR [http_cmd]: ERROR 20:20:53 DEBUG { 20:20:53 DEBUG "data": { 20:20:53 DEBUG 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG "pod": "ds-idrepo-0" 20:20:53 DEBUG 20:20:53 DEBUG "result": [ 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG "metric": { 20:20:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG ----- output ----- 20:20:53 ERROR --- status code --- 20:20:53 DEBUG "metric": { 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG 20:20:53 DEBUG { 20:20:53 DEBUG }, 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 20:20:53 DEBUG [...] 20:20:53 DEBUG 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:20:53 DEBUG "pod": "ds-idrepo-0" 20:20:53 DEBUG }, 20:20:53 DEBUG "result": [ 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG 20:20:53 DEBUG [...] 20:20:53 DEBUG [...] 20:20:53 DEBUG { 20:20:53 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:20:53 DEBUG "pod": "ds-idrepo-0" 20:20:53 DEBUG "result": [ 20:20:53 DEBUG [...] 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG { 20:20:53 DEBUG 20:20:53 DEBUG [...] 20:20:53 DEBUG 20:20:53 DEBUG [...] 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG }, 20:20:53 DEBUG "value": [ 20:20:53 DEBUG { 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG { 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG "status": "success", 20:20:53 ERROR --- http response --- 20:20:53 DEBUG }, 20:20:53 DEBUG { 20:20:53 DEBUG 20:20:53 DEBUG "data": { 20:20:53 DEBUG "metric": { 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG { 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 20:20:53 DEBUG "metric": { 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG 20:20:53 DEBUG [...] 20:20:53 DEBUG 20:20:53 DEBUG [...] 20:20:53 INFO [http_cmd]: http status code OK 20:20:53 DEBUG 20:20:53 DEBUG "data": { 20:20:53 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:20:53 DEBUG 20:20:53 DEBUG "data": { 20:20:53 DEBUG "value": [ 20:20:53 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:20:53 DEBUG "metric": { 20:20:53 DEBUG 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG "data": { 20:20:53 DEBUG 20:20:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG [...] 20:20:53 DEBUG 20:20:53 DEBUG [...] 20:20:53 DEBUG 20:20:53 DEBUG --- status code --- 20:20:53 DEBUG 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG }, 20:20:53 DEBUG 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG 20:20:53 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:20:53 DEBUG "pod": "ds-cts-0" 20:20:53 DEBUG 20:20:53 DEBUG "result": [ 20:20:53 DEBUG }, 20:20:53 DEBUG [...] 20:20:53 DEBUG 20:20:53 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG 20:20:53 DEBUG }, 20:20:53 DEBUG { 20:20:53 DEBUG 20:20:53 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:20:53 DEBUG 20:20:53 DEBUG "result": [ 20:20:53 DEBUG "value": [ 20:20:53 DEBUG "instance": "10.142.0.43:9100" 20:20:53 DEBUG "result": [ 20:20:53 DEBUG 20:20:53 DEBUG }, 20:20:53 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:20:53 DEBUG { 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 20:20:53 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:20:53 DEBUG }, 20:20:53 DEBUG "instance": "10.142.0.43:9100" 20:20:53 DEBUG }, 20:20:53 DEBUG "result": [ 20:20:53 DEBUG [...] 20:20:53 DEBUG "value": [ 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG --- http response --- 20:20:53 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:20:53 DEBUG { 20:20:53 DEBUG "metric": { 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG { 20:20:53 DEBUG 20:20:53 DEBUG [...] 20:20:53 DEBUG }, 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG }, 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 20:20:53 DEBUG "value": [ 20:20:53 DEBUG { 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG "data": { 20:20:53 DEBUG 20:20:53 DEBUG "pod": "ds-cts-2" 20:20:53 DEBUG 20:20:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:20:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:20:53 DEBUG }, 20:20:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:20:53 DEBUG }, 20:20:53 DEBUG "0" 20:20:53 DEBUG "metric": { 20:20:53 DEBUG 20:20:53 DEBUG "metric": { 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG "metric": { 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG "instance": "10.142.0.43:9100" 20:20:53 DEBUG }, 20:20:53 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:20:53 DEBUG }, 20:20:53 DEBUG ----- output ----- 20:20:53 DEBUG "value": [ 20:20:53 DEBUG }, 20:20:53 DEBUG "value": [ 20:20:53 DEBUG ] 20:20:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:20:53 DEBUG 20:20:53 DEBUG "pod": "ds-cts-0" 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG "pod": "ds-idrepo-2" 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG "0.02445614040185126" 20:20:53 DEBUG [...] 20:20:53 DEBUG "19695.13333333333" 20:20:53 DEBUG "pod": "ds-idrepo-0" 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG "result": [ 20:20:53 DEBUG }, 20:20:53 DEBUG "value": [ 20:20:53 DEBUG "value": [ 20:20:53 DEBUG "value": [ 20:20:53 DEBUG { 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG } 20:20:53 DEBUG }, 20:20:53 DEBUG 20:20:53 DEBUG }, 20:20:53 DEBUG "0" 20:20:53 DEBUG 20:20:53 DEBUG [...] 20:20:53 DEBUG }, 20:20:53 DEBUG "0.0002462942235500102" 20:20:53 DEBUG ] 20:20:53 DEBUG 20:20:53 DEBUG ] 20:20:53 DEBUG }, 20:20:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:20:53 DEBUG [...] 20:20:53 DEBUG { 20:20:53 DEBUG }, 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG "0.000333333333333341" 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG "status": "success", 20:20:53 DEBUG "0" 20:20:53 DEBUG 20:20:53 DEBUG "13121.766666666666" 20:20:53 DEBUG ] 20:20:53 DEBUG "value": [ 20:20:53 DEBUG "pod": "ds-idrepo-2" 20:20:53 DEBUG "value": [ 20:20:53 DEBUG ] 20:20:53 DEBUG [...] 20:20:53 DEBUG 20:20:53 DEBUG "value": [ 20:20:53 DEBUG ] 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG } 20:20:53 DEBUG "value": [ 20:20:53 DEBUG }, 20:20:53 DEBUG 20:20:53 DEBUG "metric": { 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG "0" 20:20:53 DEBUG "0" 20:20:53 DEBUG "data": { 20:20:53 DEBUG ] 20:20:53 DEBUG 20:20:53 DEBUG ] 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG }, 20:20:53 DEBUG 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG } 20:20:53 DEBUG ] 20:20:53 DEBUG 20:20:53 DEBUG ] 20:20:53 DEBUG 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 20:20:53 DEBUG "instance": "10.142.0.107:9100" 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG ] 20:20:53 DEBUG ] 20:20:53 DEBUG ] 20:20:53 DEBUG "resultType": "vector", 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 20:20:53 DEBUG ] 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG "0" 20:20:53 DEBUG ] 20:20:53 DEBUG } 20:20:53 DEBUG "pod": "ds-idrepo-2" 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG 20:20:53 DEBUG }, 20:20:53 DEBUG "398729216" 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG "result": [ 20:20:53 DEBUG ] 20:20:53 DEBUG [...] 20:20:53 DEBUG ] 20:20:53 DEBUG 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG ] 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG }, 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG "0" 20:20:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:20:53 DEBUG "value": [ 20:20:53 DEBUG ] 20:20:53 DEBUG ] 20:20:53 DEBUG ] 20:20:53 DEBUG ] 20:20:53 DEBUG { 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG } 20:20:53 DEBUG [...] 20:20:53 DEBUG "0" 20:20:53 DEBUG [...] 20:20:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:20:53 DEBUG "pod": "ds-cts-2" 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG "value": [ 20:20:53 DEBUG [...] 20:20:53 DEBUG ] 20:20:53 DEBUG }, 20:20:53 DEBUG "value": [ 20:20:53 DEBUG } 20:20:53 DEBUG ] 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG "metric": { 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG ] 20:20:53 DEBUG 20:20:53 DEBUG }, 20:20:53 DEBUG }, 20:20:53 DEBUG ] 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG "instance": "10.142.0.107:9100" 20:20:53 DEBUG 20:20:53 DEBUG 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG "value": [ 20:20:53 DEBUG "value": [ 20:20:53 DEBUG } 20:20:53 DEBUG "0" 20:20:53 DEBUG 20:20:53 DEBUG ] 20:20:53 DEBUG 20:20:53 DEBUG "0" 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG }, 20:20:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:20:53 DEBUG 20:20:53 DEBUG ] 20:20:53 DEBUG 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG } 20:20:53 DEBUG ] 20:20:53 DEBUG 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG ] 20:20:53 DEBUG "value": [ 20:20:53 DEBUG }, 20:20:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:20:53 DEBUG } 20:20:53 DEBUG "pod": "ds-cts-2" 20:20:53 DEBUG "0" 20:20:53 DEBUG "0.03333333333333333" 20:20:53 DEBUG } 20:20:53 DEBUG "pod": "ds-cts-2" 20:20:53 DEBUG } 20:20:53 DEBUG [...] 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG } 20:20:53 DEBUG }, 20:20:53 DEBUG ] 20:20:53 DEBUG ] 20:20:53 DEBUG ] 20:20:53 DEBUG }, 20:20:53 DEBUG 20:20:53 DEBUG ] 20:20:53 DEBUG 20:20:53 DEBUG }, 20:20:53 DEBUG "0.001" 20:20:53 DEBUG "value": [ 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG "value": [ 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG "value": [ 20:20:53 DEBUG ] 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG ] 20:20:53 DEBUG ] 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG "0" 20:20:53 DEBUG } 20:20:53 DEBUG [...] 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG } 20:20:53 DEBUG "0" 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG ] 20:20:53 DEBUG 20:20:53 DEBUG "1" 20:20:53 DEBUG ] 20:20:53 DEBUG ] 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG "instance": "10.142.0.43:9100" 20:20:53 DEBUG } 20:20:53 DEBUG 20:20:53 DEBUG ] 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG }, 20:20:53 DEBUG ] 20:20:53 DEBUG 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG ] 20:20:53 DEBUG "value": [ 20:20:53 DEBUG } 20:20:53 DEBUG "instance": "10.142.0.43:9100" 20:20:53 DEBUG ] 20:20:53 DEBUG } 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG } 20:20:53 DEBUG }, 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG "64897831936" 20:20:53 DEBUG "value": [ 20:20:53 DEBUG } 20:20:53 DEBUG ] 20:20:53 DEBUG 1713381653, 20:20:53 DEBUG } 20:20:53 DEBUG "0.000666666666666534" 20:20:53 DEBUG ] 20:20:53 DEBUG ] 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG } 20:20:53 DEBUG ] 20:20:53 DEBUG } 20:20:53 DEBUG } 20:21:53 INFO 20:21:53 INFO [loop_until]: kubectl --namespace=xlou top pods 20:21:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:21:53 INFO [loop_until]: OK (rc = 0) 20:21:53 DEBUG --- stdout --- 20:21:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 37m 969Mi am-b59c765b7-bgsxp 772m 3281Mi am-b59c765b7-ch5q4 185m 2843Mi ds-cts-0 97m 7519Mi ds-cts-1 87m 7392Mi ds-cts-2 34m 7662Mi ds-idrepo-0 36m 2986Mi ds-idrepo-1 218m 1544Mi ds-idrepo-2 235m 1545Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1008Mi lodemon-db4967764-9zt9t 195m 61Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1866m 647Mi 20:21:53 DEBUG --- stderr --- 20:21:53 DEBUG 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO 20:21:53 INFO [loop_until]: kubectl --namespace=xlou top node 20:21:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381713" 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG { 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "data": { 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "result": [ 20:21:53 DEBUG { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:21:53 DEBUG }, 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:21:53 DEBUG }, 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG "0" 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381713" 20:21:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381713" 20:21:53 ERROR [http_cmd]: ERROR 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 DEBUG --- status code --- 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 ERROR --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 DEBUG --- status code --- 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 INFO [loop_until]: OK (rc = 0) 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 ERROR --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- stdout --- 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 DEBUG --- http response --- 20:21:53 INFO [http_cmd]: http status code OK 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:21:53 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 5497Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 208m 0% 5852Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 580m 1% 5624Mi 5% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 2986Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 327m 1% 9625Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 4997Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 687m 2% 3652Mi 3% gke-xlou-cdm-ds-0f601751-cf7b 89m 0% 2875Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1651m 5% 5215Mi 4% gke-xlou-cdm-ds-0f601751-j488 309m 1% 4027Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 180m 0% 10753Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 163m 0% 4553Mi 4% gke-xlou-cdm-ds-0f601751-nmxm 189m 0% 9230Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 2607m 16% 4522Mi 7% 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:21:53 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:21:53 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:21:53 DEBUG --- status code --- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG --- stderr --- 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG { 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:21:53 DEBUG { 20:21:53 DEBUG 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG { 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG { 20:21:53 DEBUG --- http response --- 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG { 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "data": { 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG "data": { 20:21:53 DEBUG "data": { 20:21:53 DEBUG "data": { 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:21:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:21:53 DEBUG "data": { 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "data": { 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "data": { 20:21:53 DEBUG "data": { 20:21:53 DEBUG "data": { 20:21:53 DEBUG "data": { 20:21:53 DEBUG "data": { 20:21:53 DEBUG "data": { 20:21:53 DEBUG "data": { 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "data": { 20:21:53 DEBUG "data": { 20:21:53 DEBUG "data": { 20:21:53 DEBUG { 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "data": { 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG ----- output ----- 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "data": { 20:21:53 DEBUG "data": { 20:21:53 DEBUG "data": { 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "data": { 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "result": [ 20:21:53 DEBUG { 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "data": { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG "status": "success", 20:21:53 DEBUG { 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "result": [ 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG "result": [ 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG { 20:21:53 DEBUG "data": { 20:21:53 DEBUG "data": { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG { 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:21:53 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:21:53 DEBUG "instance": "10.142.0.107:9100" 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "resultType": "vector", 20:21:53 DEBUG "pod": "ds-cts-0" 20:21:53 DEBUG "metric": { 20:21:53 DEBUG { 20:21:53 DEBUG "pod": "ds-idrepo-0" 20:21:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:21:53 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:21:53 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:21:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:21:53 DEBUG "instance": "10.142.0.107:9100" 20:21:53 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:21:53 DEBUG "pod": "ds-idrepo-0" 20:21:53 DEBUG }, 20:21:53 DEBUG "pod": "ds-cts-0" 20:21:53 DEBUG "pod": "ds-idrepo-0" 20:21:53 DEBUG "pod": "ds-cts-0" 20:21:53 DEBUG { 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG "pod": "ds-idrepo-0" 20:21:53 DEBUG "result": [ 20:21:53 DEBUG "result": [ 20:21:53 DEBUG }, 20:21:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:21:53 DEBUG "metric": { 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG "value": [ 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG }, 20:21:53 DEBUG { 20:21:53 DEBUG { 20:21:53 DEBUG "value": [ 20:21:53 DEBUG }, 20:21:53 DEBUG "instance": "10.142.0.107:9100" 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG }, 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "metric": { 20:21:53 DEBUG "metric": { 20:21:53 DEBUG 20:21:53 DEBUG "value": [ 20:21:53 DEBUG }, 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG "instance": "10.142.0.107:9100" 20:21:53 DEBUG "instance": "10.142.0.107:9100" 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG }, 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG [...] 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG [...] 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG [...] 20:21:53 DEBUG [...] 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG "pod": "ds-cts-0" 20:21:53 DEBUG 20:21:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG "pod": "ds-cts-0" 20:21:53 DEBUG }, 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:21:53 DEBUG }, 20:21:53 DEBUG "instance": "10.142.0.43:9100" 20:21:53 DEBUG 20:21:53 DEBUG [...] 20:21:53 DEBUG [...] 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG }, 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG }, 20:21:53 DEBUG 20:21:53 DEBUG }, 20:21:53 DEBUG "instance": "10.142.0.43:9100" 20:21:53 DEBUG "pod": "ds-idrepo-2" 20:21:53 DEBUG "pod": "ds-cts-2" 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 20:21:53 DEBUG "pod": "ds-idrepo-2" 20:21:53 DEBUG "pod": "ds-cts-2" 20:21:53 DEBUG "pod": "ds-cts-0" 20:21:53 DEBUG "value": [ 20:21:53 DEBUG }, 20:21:53 DEBUG "pod": "ds-idrepo-2" 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG "pod": "ds-cts-2" 20:21:53 DEBUG "pod": "ds-cts-2" 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:21:53 DEBUG }, 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "pod": "ds-cts-0" 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:21:53 DEBUG "value": [ 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG }, 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG 20:21:53 DEBUG }, 20:21:53 DEBUG "value": [ 20:21:53 DEBUG }, 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG "0.001" 20:21:53 DEBUG ] 20:21:53 DEBUG 20:21:53 DEBUG 20:21:53 DEBUG }, 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG "0" 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG "531.2348000000001" 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG "237139.94106231583" 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "0" 20:21:53 DEBUG 20:21:53 DEBUG }, 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "value": [ 20:21:53 DEBUG }, 20:21:53 DEBUG } 20:21:53 DEBUG 20:21:53 DEBUG "instance": "10.142.0.43:9100" 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 20:21:53 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:21:53 DEBUG ] 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG }, 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG }, 20:21:53 DEBUG "832.2666666666667" 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG ] 20:21:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "497.8953194444445" 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG "value": [ 20:21:53 DEBUG ] 20:21:53 DEBUG "instance": "10.142.0.43:9100" 20:21:53 DEBUG }, 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG "0" 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG "530.9155583333333" 20:21:53 DEBUG "0" 20:21:53 DEBUG "value": [ 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG "value": [ 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG "524.3953194444445" 20:21:53 DEBUG "0" 20:21:53 DEBUG } 20:21:53 DEBUG }, 20:21:53 DEBUG "value": [ 20:21:53 DEBUG ] 20:21:53 DEBUG "0" 20:21:53 DEBUG "13919556.333333332" 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG } 20:21:53 DEBUG }, 20:21:53 DEBUG "value": [ 20:21:53 DEBUG "instance": "10.142.0.43:9100" 20:21:53 DEBUG }, 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG "13912167.033333333" 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG "498.82865277777773" 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG }, 20:21:53 DEBUG "value": [ 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG "0.4529754905484053" 20:21:53 DEBUG } 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG "3598904.046575146" 20:21:53 DEBUG "12467" 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG "0.018333333333332764" 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG "834031616" 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG "65053522944" 20:21:53 DEBUG "value": [ 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG "2.1901142289960673" 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG 1713381713, 20:21:53 DEBUG ] 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG "0.012666666666666632" 20:21:53 DEBUG } 20:21:53 DEBUG } 20:21:53 DEBUG ] 20:21:53 DEBUG } 20:21:54 DEBUG } 20:21:54 DEBUG ] 20:21:54 DEBUG ] 20:21:54 DEBUG } 20:21:54 DEBUG } 20:21:54 DEBUG } 20:21:54 DEBUG } 20:21:54 DEBUG } 20:21:54 DEBUG ] 20:21:54 DEBUG } 20:21:54 DEBUG } 20:21:54 DEBUG } 20:22:53 INFO 20:22:53 INFO [loop_until]: kubectl --namespace=xlou top pods 20:22:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:22:53 INFO [loop_until]: OK (rc = 0) 20:22:53 DEBUG --- stdout --- 20:22:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 9394m 4553Mi am-b59c765b7-bgsxp 8383m 4430Mi am-b59c765b7-ch5q4 9630m 4532Mi ds-cts-0 4879m 7560Mi ds-cts-1 4838m 7431Mi ds-cts-2 5124m 7683Mi ds-idrepo-0 4776m 2990Mi ds-idrepo-1 4878m 1903Mi ds-idrepo-2 4735m 1932Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 965Mi idm-b85f9d978-wnhxb 4m 1008Mi lodemon-db4967764-9zt9t 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6864m 779Mi 20:22:53 DEBUG --- stderr --- 20:22:53 DEBUG 20:22:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381773" 20:22:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713381773" 20:22:53 INFO [http_cmd]: http status code OK 20:22:53 DEBUG --- status code --- 20:22:53 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:53 DEBUG --- http response --- 20:22:53 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:22:53 DEBUG ----- output ----- 20:22:53 DEBUG { 20:22:53 DEBUG "status": "success", 20:22:53 DEBUG "data": { 20:22:53 DEBUG "resultType": "vector", 20:22:53 DEBUG "result": [ 20:22:53 DEBUG { 20:22:53 DEBUG "metric": { 20:22:53 DEBUG "pod": "am-b59c765b7-ch5q4" 20:22:53 DEBUG }, 20:22:53 DEBUG "value": [ 20:22:53 DEBUG 20:22:53 DEBUG 20:22:53 DEBUG 20:22:53 DEBUG [...] 20:22:53 DEBUG 20:22:53 DEBUG 20:22:53 DEBUG 20:22:53 DEBUG "pod": "am-b59c765b7-bgsxp" 20:22:53 DEBUG }, 20:22:53 DEBUG "value": [ 20:22:53 DEBUG 1713381773, 20:22:53 DEBUG "0" 20:22:53 DEBUG ] 20:22:53 DEBUG } 20:22:53 DEBUG ] 20:22:53 DEBUG } 20:22:53 DEBUG } 20:22:53 ERROR [http_cmd]: ERROR 20:22:53 ERROR --- status code --- 20:22:53 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:22:53 ERROR --- http response --- 20:22:53 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:22:53 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:22:53 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713381773" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381774" 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "data": { 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "result": [ 20:22:54 DEBUG { 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:22:54 DEBUG }, 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:22:54 DEBUG }, 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 1713381773, 20:22:54 DEBUG "0.0015" 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "data": { 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "result": [ 20:22:54 DEBUG { 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "instance": "10.142.0.107:9100" 20:22:54 DEBUG }, 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "instance": "10.142.0.43:9100" 20:22:54 DEBUG }, 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG "value": [ 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG "0.04026666666666614" 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:22:54 INFO 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG ----- output ----- 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 INFO [loop_until]: kubectl --namespace=xlou top node 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG { 20:22:54 DEBUG --- status code --- 20:22:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG "data": { 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:22:54 DEBUG "result": [ 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG { 20:22:54 DEBUG { 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:22:54 DEBUG "data": { 20:22:54 DEBUG }, 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "value": [ 20:22:54 DEBUG "result": [ 20:22:54 DEBUG 20:22:54 DEBUG { 20:22:54 DEBUG 20:22:54 DEBUG "metric": { 20:22:54 DEBUG 20:22:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:22:54 DEBUG [...] 20:22:54 DEBUG }, 20:22:54 DEBUG 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:22:54 DEBUG 20:22:54 DEBUG }, 20:22:54 DEBUG [...] 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG 20:22:54 DEBUG "914505728" 20:22:54 DEBUG 20:22:54 DEBUG ] 20:22:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:22:54 DEBUG } 20:22:54 DEBUG }, 20:22:54 DEBUG ] 20:22:54 DEBUG "value": [ 20:22:54 DEBUG } 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG } 20:22:54 DEBUG "1822.8666666666666" 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713381774" 20:22:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381774" 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG --- status code --- 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG --- status code --- 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG --- status code --- 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG --- status code --- 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG --- status code --- 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:22:54 DEBUG ----- output ----- 20:22:54 INFO [loop_until]: OK (rc = 0) 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:22:54 DEBUG ----- output ----- 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG { 20:22:54 DEBUG --- stdout --- 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:22:54 INFO [http_cmd]: http status code OK 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG { 20:22:54 DEBUG { 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG { 20:22:54 DEBUG { 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG { 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG { 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:22:54 DEBUG { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG { 20:22:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 5504Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4463m 14% 5870Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9521m 31% 7142Mi 6% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 2985Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 5084m 17% 9753Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 5008Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9452m 31% 7243Mi 6% gke-xlou-cdm-ds-0f601751-cf7b 27501m 92% 2915Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9242m 30% 6404Mi 5% gke-xlou-cdm-ds-0f601751-j488 5047m 16% 4439Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 4922m 16% 10850Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5368m 17% 4949Mi 4% gke-xlou-cdm-ds-0f601751-nmxm 5149m 17% 9340Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6954m 43% 4595Mi 7% 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG --- status code --- 20:22:54 DEBUG { 20:22:54 DEBUG { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG { 20:22:54 DEBUG { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "data": { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG --- stderr --- 20:22:54 DEBUG { 20:22:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "data": { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "data": { 20:22:54 DEBUG "data": { 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:22:54 DEBUG "data": { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "data": { 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "data": { 20:22:54 DEBUG { 20:22:54 DEBUG "data": { 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "data": { 20:22:54 DEBUG 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG --- http response --- 20:22:54 DEBUG "data": { 20:22:54 DEBUG "data": { 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "data": { 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "data": { 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:22:54 DEBUG "data": { 20:22:54 DEBUG "data": { 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "result": [ 20:22:54 DEBUG { 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "result": [ 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "result": [ 20:22:54 DEBUG "result": [ 20:22:54 DEBUG { 20:22:54 DEBUG "result": [ 20:22:54 DEBUG "result": [ 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "result": [ 20:22:54 DEBUG "data": { 20:22:54 DEBUG "result": [ 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "data": { 20:22:54 DEBUG "result": [ 20:22:54 DEBUG ----- output ----- 20:22:54 DEBUG "result": [ 20:22:54 DEBUG "result": [ 20:22:54 DEBUG { 20:22:54 DEBUG "result": [ 20:22:54 DEBUG { 20:22:54 DEBUG { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG { 20:22:54 DEBUG { 20:22:54 DEBUG "result": [ 20:22:54 DEBUG { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "result": [ 20:22:54 DEBUG { 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG { 20:22:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG { 20:22:54 DEBUG { 20:22:54 DEBUG { 20:22:54 DEBUG { 20:22:54 DEBUG "metric": { 20:22:54 DEBUG { 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "data": { 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "metric": { 20:22:54 DEBUG { 20:22:54 DEBUG "result": [ 20:22:54 DEBUG "data": { 20:22:54 DEBUG { 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "result": [ 20:22:54 DEBUG "metric": { 20:22:54 DEBUG }, 20:22:54 DEBUG "result": [ 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "status": "success", 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "instance": "10.142.0.107:9100" 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "instance": "10.142.0.107:9100" 20:22:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "pod": "ds-idrepo-0" 20:22:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:22:54 DEBUG "metric": { 20:22:54 DEBUG { 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:22:54 DEBUG { 20:22:54 DEBUG "instance": "10.142.0.107:9100" 20:22:54 DEBUG "value": [ 20:22:54 DEBUG { 20:22:54 DEBUG "pod": "ds-idrepo-0" 20:22:54 DEBUG "data": { 20:22:54 DEBUG "pod": "ds-idrepo-0" 20:22:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:22:54 DEBUG }, 20:22:54 DEBUG "pod": "ds-idrepo-0" 20:22:54 DEBUG }, 20:22:54 DEBUG }, 20:22:54 DEBUG "result": [ 20:22:54 DEBUG }, 20:22:54 DEBUG }, 20:22:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "result": [ 20:22:54 DEBUG "pod": "ds-cts-0" 20:22:54 DEBUG }, 20:22:54 DEBUG "metric": { 20:22:54 DEBUG }, 20:22:54 DEBUG 20:22:54 DEBUG "metric": { 20:22:54 DEBUG }, 20:22:54 DEBUG "resultType": "vector", 20:22:54 DEBUG }, 20:22:54 DEBUG }, 20:22:54 DEBUG "value": [ 20:22:54 DEBUG }, 20:22:54 DEBUG "value": [ 20:22:54 DEBUG "value": [ 20:22:54 DEBUG { 20:22:54 DEBUG "value": [ 20:22:54 DEBUG "value": [ 20:22:54 DEBUG }, 20:22:54 DEBUG "pod": "ds-cts-0" 20:22:54 DEBUG { 20:22:54 DEBUG }, 20:22:54 DEBUG "value": [ 20:22:54 DEBUG "pod": "ds-cts-0" 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:22:54 DEBUG "value": [ 20:22:54 DEBUG "result": [ 20:22:54 DEBUG "value": [ 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "metric": { 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "value": [ 20:22:54 DEBUG }, 20:22:54 DEBUG "metric": { 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG }, 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG }, 20:22:54 DEBUG 20:22:54 DEBUG { 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "value": [ 20:22:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG "metric": { 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG }, 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG }, 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "instance": "10.142.0.107:9100" 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "value": [ 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG }, 20:22:54 DEBUG [...] 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "pod": "ds-cts-2" 20:22:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "pod": "ds-idrepo-2" 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:22:54 DEBUG }, 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG }, 20:22:54 DEBUG }, 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "instance": "10.142.0.43:9100" 20:22:54 DEBUG 20:22:54 DEBUG "instance": "10.142.0.43:9100" 20:22:54 DEBUG 20:22:54 DEBUG }, 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG [...] 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "value": [ 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG "pod": "ds-idrepo-2" 20:22:54 DEBUG 20:22:54 DEBUG "pod": "ds-idrepo-2" 20:22:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:22:54 DEBUG }, 20:22:54 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:22:54 DEBUG }, 20:22:54 DEBUG "pod": "ds-cts-2" 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:22:54 DEBUG "instance": "10.142.0.43:9100" 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG 20:22:54 DEBUG }, 20:22:54 DEBUG [...] 20:22:54 DEBUG }, 20:22:54 DEBUG }, 20:22:54 DEBUG "value": [ 20:22:54 DEBUG }, 20:22:54 DEBUG "value": [ 20:22:54 DEBUG }, 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG 20:22:54 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:22:54 DEBUG "0" 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG 20:22:54 DEBUG }, 20:22:54 DEBUG }, 20:22:54 DEBUG "0" 20:22:54 DEBUG "0" 20:22:54 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 20:22:54 DEBUG "value": [ 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG "value": [ 20:22:54 DEBUG "1.1" 20:22:54 DEBUG 20:22:54 DEBUG }, 20:22:54 DEBUG ] 20:22:54 DEBUG "pod": "ds-cts-2" 20:22:54 DEBUG 20:22:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:22:54 DEBUG "value": [ 20:22:54 DEBUG "value": [ 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG }, 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG "48130007.43333333" 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG "66729260032" 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG ] 20:22:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:22:54 DEBUG "value": [ 20:22:54 DEBUG } 20:22:54 DEBUG }, 20:22:54 DEBUG "pod": "ds-cts-2" 20:22:54 DEBUG }, 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG ] 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG "1.1" 20:22:54 DEBUG ] 20:22:54 DEBUG "1.1" 20:22:54 DEBUG "1822.7" 20:22:54 DEBUG ] 20:22:54 DEBUG "0" 20:22:54 DEBUG ] 20:22:54 DEBUG "0" 20:22:54 DEBUG } 20:22:54 DEBUG }, 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG }, 20:22:54 DEBUG "value": [ 20:22:54 DEBUG "1822.5666666666666" 20:22:54 DEBUG "48136218.7" 20:22:54 DEBUG } 20:22:54 DEBUG "value": [ 20:22:54 DEBUG } 20:22:54 DEBUG 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG ] 20:22:54 DEBUG "value": [ 20:22:54 DEBUG "0" 20:22:54 DEBUG "value": [ 20:22:54 DEBUG } 20:22:54 DEBUG "value": [ 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG ] 20:22:54 DEBUG ] 20:22:54 DEBUG ] 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG "0.00004386576530148167" 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG ] 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG "6.7916382727740245" 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG "instance": "10.142.0.43:9100" 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG "106987.5" 20:22:54 DEBUG } 20:22:54 DEBUG "0" 20:22:54 DEBUG "0" 20:22:54 DEBUG ] 20:22:54 DEBUG ] 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG }, 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG ] 20:22:54 DEBUG ] 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG "value": [ 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG 1713381774, 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG "0.027999999999999935" 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG } 20:22:54 DEBUG ] 20:22:54 DEBUG } 20:22:54 DEBUG } 20:23:53 INFO 20:23:53 INFO [loop_until]: kubectl --namespace=xlou top pods 20:23:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:23:53 INFO [loop_until]: OK (rc = 0) 20:23:53 DEBUG --- stdout --- 20:23:53 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 9376m 4785Mi am-b59c765b7-bgsxp 8468m 4662Mi am-b59c765b7-ch5q4 9598m 4682Mi ds-cts-0 4938m 7571Mi ds-cts-1 4855m 7451Mi ds-cts-2 5103m 7733Mi ds-idrepo-0 4680m 2997Mi ds-idrepo-1 4864m 2143Mi ds-idrepo-2 4716m 2064Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1008Mi lodemon-db4967764-9zt9t 10m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6691m 783Mi 20:23:53 DEBUG --- stderr --- 20:23:53 DEBUG 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713381834" 20:23:54 ERROR [http_cmd]: ERROR 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 ERROR --- status code --- 20:23:54 DEBUG --- status code --- 20:23:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 ERROR --- http response --- 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:23:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG { 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG "data": { 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "result": [ 20:23:54 DEBUG { 20:23:54 DEBUG "metric": { 20:23:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG [...] 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG "0.002" 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG { 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG "data": { 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "result": [ 20:23:54 DEBUG { 20:23:54 DEBUG "metric": { 20:23:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG [...] 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG "0" 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG { 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG "data": { 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "result": [ 20:23:54 DEBUG { 20:23:54 DEBUG "metric": { 20:23:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG [...] 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG "916975616" 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG { 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG "data": { 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "result": [ 20:23:54 DEBUG { 20:23:54 DEBUG "metric": { 20:23:54 DEBUG "instance": "10.142.0.107:9100" 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG [...] 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "instance": "10.142.0.43:9100" 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG "0.041866666666665726" 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG { 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG "data": { 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "result": [ 20:23:54 DEBUG { 20:23:54 DEBUG "metric": { 20:23:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG [...] 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG "1821.8666666666666" 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381834" 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 DEBUG --- status code --- 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381834" 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG { 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG "data": { 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "result": [ 20:23:54 DEBUG { 20:23:54 DEBUG "metric": { 20:23:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG [...] 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG "0" 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG { 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG "data": { 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:23:54 DEBUG "resultType": "vector", 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG ----- output ----- 20:23:54 INFO 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG "result": [ 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG { 20:23:54 INFO [loop_until]: kubectl --namespace=xlou top node 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG { 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG "data": { 20:23:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:23:54 DEBUG "metric": { 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "pod": "ds-cts-0" 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:23:54 DEBUG "result": [ 20:23:54 DEBUG }, 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:23:54 DEBUG { 20:23:54 DEBUG "value": [ 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG { 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG "metric": { 20:23:54 DEBUG 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG { 20:23:54 DEBUG "instance": "10.142.0.107:9100" 20:23:54 DEBUG 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG "data": { 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG }, 20:23:54 DEBUG 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:23:54 DEBUG "data": { 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG "value": [ 20:23:54 DEBUG [...] 20:23:54 DEBUG ----- output ----- 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG "result": [ 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG { 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG { 20:23:54 DEBUG { 20:23:54 DEBUG "result": [ 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG "metric": { 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG { 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "data": { 20:23:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:23:54 DEBUG "data": { 20:23:54 DEBUG "metric": { 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:23:54 DEBUG [...] 20:23:54 DEBUG "pod": "ds-cts-2" 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG }, 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "instance": "10.142.0.107:9100" 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG 20:23:54 DEBUG }, 20:23:54 DEBUG "result": [ 20:23:54 DEBUG "value": [ 20:23:54 DEBUG { 20:23:54 DEBUG "result": [ 20:23:54 DEBUG }, 20:23:54 DEBUG { 20:23:54 DEBUG 20:23:54 DEBUG "value": [ 20:23:54 DEBUG { 20:23:54 DEBUG 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG { 20:23:54 DEBUG "value": [ 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG "metric": { 20:23:54 DEBUG 20:23:54 DEBUG "data": { 20:23:54 DEBUG "metric": { 20:23:54 DEBUG 20:23:54 DEBUG "data": { 20:23:54 DEBUG "instance": "10.142.0.43:9100" 20:23:54 DEBUG "0" 20:23:54 DEBUG "pod": "ds-idrepo-0" 20:23:54 DEBUG 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "pod": "ds-idrepo-0" 20:23:54 DEBUG 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG }, 20:23:54 DEBUG ] 20:23:54 DEBUG }, 20:23:54 DEBUG [...] 20:23:54 DEBUG "result": [ 20:23:54 DEBUG }, 20:23:54 DEBUG 20:23:54 DEBUG "result": [ 20:23:54 DEBUG "value": [ 20:23:54 DEBUG } 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 20:23:54 DEBUG { 20:23:54 DEBUG "value": [ 20:23:54 DEBUG [...] 20:23:54 DEBUG { 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG ] 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "metric": { 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "metric": { 20:23:54 DEBUG "48059560.36666667" 20:23:54 DEBUG } 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG "instance": "10.142.0.107:9100" 20:23:54 DEBUG 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 DEBUG 20:23:54 DEBUG "instance": "10.142.0.107:9100" 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG 20:23:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG }, 20:23:54 DEBUG 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG 20:23:54 DEBUG }, 20:23:54 DEBUG } 20:23:54 DEBUG [...] 20:23:54 DEBUG }, 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG "value": [ 20:23:54 DEBUG [...] 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG "instance": "10.142.0.43:9100" 20:23:54 DEBUG "value": [ 20:23:54 DEBUG ] 20:23:54 DEBUG 20:23:54 DEBUG "value": [ 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG }, 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 DEBUG 20:23:54 DEBUG } 20:23:54 DEBUG 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "value": [ 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:23:54 DEBUG 20:23:54 DEBUG } 20:23:54 DEBUG 20:23:54 DEBUG "6.758005687023797" 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG 20:23:54 DEBUG [...] 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG ----- output ----- 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 DEBUG 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 DEBUG "pod": "ds-cts-2" 20:23:54 DEBUG }, 20:23:54 DEBUG { 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "69090823168" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 DEBUG { 20:23:54 DEBUG [...] 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381834" 20:23:54 DEBUG ] 20:23:54 DEBUG "value": [ 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG "pod": "ds-idrepo-2" 20:23:54 DEBUG 20:23:54 DEBUG ] 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG } 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG "data": { 20:23:54 DEBUG }, 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG 20:23:54 DEBUG } 20:23:54 DEBUG "data": { 20:23:54 DEBUG 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG ] 20:23:54 DEBUG "0" 20:23:54 DEBUG "resultType": "vector", 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG "value": [ 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG "instance": "10.142.0.43:9100" 20:23:54 DEBUG ] 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG "result": [ 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG }, 20:23:54 DEBUG } 20:23:54 DEBUG "result": [ 20:23:54 DEBUG "instance": "10.142.0.43:9100" 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG { 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG "0" 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG "value": [ 20:23:54 DEBUG } 20:23:54 DEBUG { 20:23:54 DEBUG }, 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:23:54 DEBUG ] 20:23:54 DEBUG "metric": { 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG ] 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG "metric": { 20:23:54 DEBUG "value": [ 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG } 20:23:54 DEBUG "pod": "ds-idrepo-0" 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:23:54 DEBUG } 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG ----- output ----- 20:23:54 INFO [http_cmd]: http status code OK 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:23:54 DEBUG "48053138.233333334" 20:23:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG { 20:23:54 DEBUG { 20:23:54 DEBUG } 20:23:54 DEBUG }, 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG ] 20:23:54 DEBUG { 20:23:54 DEBUG { 20:23:54 DEBUG --- status code --- 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG ] 20:23:54 DEBUG }, 20:23:54 DEBUG "0.029666666666666574" 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG "value": [ 20:23:54 DEBUG { 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:23:54 DEBUG { 20:23:54 DEBUG } 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:23:54 DEBUG { 20:23:54 DEBUG } 20:23:54 DEBUG "value": [ 20:23:54 DEBUG ] 20:23:54 DEBUG "data": { 20:23:54 DEBUG "data": { 20:23:54 DEBUG 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG } 20:23:54 DEBUG "data": { 20:23:54 DEBUG "data": { 20:23:54 DEBUG --- http response --- 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG ] 20:23:54 DEBUG 20:23:54 DEBUG } 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG 20:23:54 DEBUG "data": { 20:23:54 DEBUG { 20:23:54 DEBUG "data": { 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "data": { 20:23:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:23:54 DEBUG } 20:23:54 DEBUG 20:23:54 DEBUG ] 20:23:54 DEBUG "result": [ 20:23:54 DEBUG "result": [ 20:23:54 DEBUG 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG "result": [ 20:23:54 DEBUG "result": [ 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG ----- output ----- 20:23:54 DEBUG } 20:23:54 DEBUG 20:23:54 DEBUG } 20:23:54 DEBUG { 20:23:54 DEBUG { 20:23:54 DEBUG [...] 20:23:54 DEBUG "result": [ 20:23:54 DEBUG "data": { 20:23:54 DEBUG "result": [ 20:23:54 DEBUG { 20:23:54 DEBUG { 20:23:54 DEBUG "result": [ 20:23:54 DEBUG { 20:23:54 DEBUG [...] 20:23:54 DEBUG } 20:23:54 DEBUG "metric": { 20:23:54 DEBUG "metric": { 20:23:54 DEBUG 20:23:54 DEBUG { 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG { 20:23:54 DEBUG "metric": { 20:23:54 DEBUG "metric": { 20:23:54 DEBUG { 20:23:54 DEBUG "status": "success", 20:23:54 DEBUG "data": { 20:23:54 DEBUG "pod": "ds-idrepo-0" 20:23:54 DEBUG "pod": "ds-cts-0" 20:23:54 DEBUG 20:23:54 DEBUG "metric": { 20:23:54 DEBUG "result": [ 20:23:54 DEBUG "metric": { 20:23:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:23:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:23:54 DEBUG "metric": { 20:23:54 DEBUG 20:23:54 DEBUG "resultType": "vector", 20:23:54 DEBUG }, 20:23:54 DEBUG }, 20:23:54 DEBUG 20:23:54 DEBUG "pod": "ds-cts-0" 20:23:54 DEBUG { 20:23:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:23:54 DEBUG }, 20:23:54 DEBUG }, 20:23:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:23:54 DEBUG 20:23:54 DEBUG "result": [ 20:23:54 DEBUG "value": [ 20:23:54 DEBUG "value": [ 20:23:54 DEBUG "pod": "ds-idrepo-2" 20:23:54 DEBUG }, 20:23:54 DEBUG "metric": { 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG "value": [ 20:23:54 DEBUG }, 20:23:54 DEBUG 20:23:54 DEBUG { 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "value": [ 20:23:54 DEBUG "pod": "ds-idrepo-0" 20:23:54 DEBUG "metric": { 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 20:23:54 DEBUG }, 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG }, 20:23:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "value": [ 20:23:54 DEBUG }, 20:23:54 DEBUG [...] 20:23:54 DEBUG [...] 20:23:54 DEBUG "0" 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG [...] 20:23:54 DEBUG [...] 20:23:54 DEBUG 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG ] 20:23:54 DEBUG [...] 20:23:54 DEBUG 20:23:54 DEBUG [...] 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG [...] 20:23:54 DEBUG "2268.069711068987" 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG } 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG ] 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG ] 20:23:54 DEBUG 20:23:54 DEBUG [...] 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG } 20:23:54 DEBUG 20:23:54 DEBUG "pod": "ds-idrepo-2" 20:23:54 DEBUG "pod": "ds-cts-2" 20:23:54 DEBUG } 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG 20:23:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:23:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:23:54 DEBUG 20:23:54 DEBUG ] 20:23:54 DEBUG [...] 20:23:54 DEBUG }, 20:23:54 DEBUG }, 20:23:54 DEBUG } 20:23:54 DEBUG "pod": "ds-cts-2" 20:23:54 DEBUG 20:23:54 DEBUG "pod": "ds-idrepo-0" 20:23:54 DEBUG }, 20:23:54 DEBUG }, 20:23:54 DEBUG "pod": "ds-idrepo-0" 20:23:54 DEBUG } 20:23:54 DEBUG 20:23:54 DEBUG "value": [ 20:23:54 DEBUG "value": [ 20:23:54 DEBUG }, 20:23:54 DEBUG 20:23:54 DEBUG }, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG "value": [ 20:23:54 DEBUG }, 20:23:54 DEBUG } 20:23:54 DEBUG 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:23:54 DEBUG }, 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG "value": [ 20:23:54 DEBUG 20:23:54 DEBUG "0" 20:23:54 DEBUG "0" 20:23:54 DEBUG ] 20:23:54 DEBUG "value": [ 20:23:54 DEBUG "value": [ 20:23:54 DEBUG "1821.8666666666666" 20:23:54 DEBUG "1821.8333333333333" 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG "pod": "ds-idrepo-0" 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG ] 20:23:54 DEBUG ] 20:23:54 DEBUG "21340.474099603653" 20:23:54 DEBUG }, 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG "0" 20:23:54 DEBUG "18453.839921879488" 20:23:54 DEBUG "215903.5" 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG "value": [ 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG ] 20:23:54 DEBUG ] 20:23:54 DEBUG ] 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG 1713381834, 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG "2.0481589686139485" 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG ] 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG ] 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 DEBUG } 20:23:54 INFO [loop_until]: OK (rc = 0) 20:23:54 DEBUG --- stdout --- 20:23:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 5512Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4565m 15% 5857Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9439m 31% 7249Mi 6% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 2989Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 5087m 17% 9786Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5007Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9524m 31% 7384Mi 6% gke-xlou-cdm-ds-0f601751-cf7b 27766m 93% 2907Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 8844m 29% 6489Mi 5% gke-xlou-cdm-ds-0f601751-j488 4955m 16% 4631Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 4812m 16% 10866Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5182m 17% 5163Mi 4% gke-xlou-cdm-ds-0f601751-nmxm 5167m 17% 9382Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6901m 43% 4585Mi 7% 20:23:54 DEBUG --- stderr --- 20:23:54 DEBUG 20:24:54 INFO 20:24:54 INFO [loop_until]: kubectl --namespace=xlou top pods 20:24:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713381894" 20:24:54 INFO [loop_until]: OK (rc = 0) 20:24:54 DEBUG --- stdout --- 20:24:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 9447m 7175Mi am-b59c765b7-bgsxp 8686m 7110Mi am-b59c765b7-ch5q4 9525m 7022Mi ds-cts-0 5373m 7608Mi ds-cts-1 4987m 7497Mi ds-cts-2 5606m 7771Mi ds-idrepo-0 4600m 3066Mi ds-idrepo-1 4739m 2316Mi ds-idrepo-2 4687m 2259Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1008Mi lodemon-db4967764-9zt9t 1m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6631m 818Mi 20:24:54 DEBUG --- stderr --- 20:24:54 DEBUG 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "data": { 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG "954691584" 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713381894" 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381894" 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 ERROR [http_cmd]: ERROR 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 ERROR --- status code --- 20:24:54 DEBUG --- status code --- 20:24:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 ERROR --- http response --- 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:24:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "data": { 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG "0.002" 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG { 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "data": { 20:24:54 DEBUG "data": { 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "result": [ 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:24:54 DEBUG "instance": "10.142.0.107:9100" 20:24:54 DEBUG }, 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG [...] 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:24:54 DEBUG "instance": "10.142.0.43:9100" 20:24:54 DEBUG }, 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG "544.79573" 20:24:54 DEBUG "0.04593333333333333" 20:24:54 DEBUG ] 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "data": { 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG "1815.3" 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381894" 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "data": { 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "instance": "10.142.0.107:9100" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "instance": "10.142.0.43:9100" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG "51496264" 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "data": { 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG "0" 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "data": { 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "pod": "ds-cts-0" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "pod": "ds-cts-2" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG "0.000050875555555555556" 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381894" 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "data": { 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG "1815.1666666666667" 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "data": { 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "instance": "10.142.0.107:9100" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "instance": "10.142.0.43:9100" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG "51490631.03333333" 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713381894" 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381894" 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG --- http response --- 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG { 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "data": { 20:24:54 DEBUG "data": { 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "result": [ 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:24:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:24:54 DEBUG }, 20:24:54 DEBUG }, 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381894" 20:24:54 DEBUG "value": [ 20:24:54 INFO 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 INFO [loop_until]: kubectl --namespace=xlou top node 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:24:54 DEBUG 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG [...] 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG [...] 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 DEBUG 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 DEBUG 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:24:54 DEBUG --- status code --- 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:24:54 DEBUG { 20:24:54 DEBUG { 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381894" 20:24:54 DEBUG }, 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG }, 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG --- http response --- 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG "value": [ 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG "value": [ 20:24:54 DEBUG "data": { 20:24:54 DEBUG "data": { 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:24:54 DEBUG ----- output ----- 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG "0" 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG "0" 20:24:54 DEBUG "result": [ 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG ] 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG --- http response --- 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG "data": { 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 INFO [http_cmd]: http status code OK 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG ] 20:24:54 DEBUG { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG { 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:24:54 DEBUG } 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG --- status code --- 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG } 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "instance": "10.142.0.107:9100" 20:24:54 DEBUG "metric": { 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG ] 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:24:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:24:54 DEBUG ] 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:24:54 DEBUG "data": { 20:24:54 DEBUG }, 20:24:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:24:54 DEBUG { 20:24:54 DEBUG } 20:24:54 DEBUG { 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG } 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG } 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG --- http response --- 20:24:54 DEBUG "data": { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "data": { 20:24:54 DEBUG { 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:24:54 DEBUG } 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:24:54 DEBUG { 20:24:54 DEBUG "result": [ 20:24:54 DEBUG }, 20:24:54 DEBUG 20:24:54 DEBUG "data": { 20:24:54 DEBUG "data": { 20:24:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "pod": "ds-idrepo-0" 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG { 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG ----- output ----- 20:24:54 DEBUG "result": [ 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG { 20:24:54 DEBUG "data": { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG "result": [ 20:24:54 DEBUG "result": [ 20:24:54 DEBUG { 20:24:54 DEBUG { 20:24:54 DEBUG "data": { 20:24:54 DEBUG 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "pod": "ds-idrepo-0" 20:24:54 DEBUG }, 20:24:54 DEBUG 20:24:54 DEBUG { 20:24:54 DEBUG { 20:24:54 DEBUG "status": "success", 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "data": { 20:24:54 DEBUG "data": { 20:24:54 DEBUG "result": [ 20:24:54 DEBUG 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "metric": { 20:24:54 DEBUG "data": { 20:24:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:24:54 DEBUG "result": [ 20:24:54 DEBUG 20:24:54 DEBUG "instance": "10.142.0.107:9100" 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG { 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "pod": "ds-idrepo-0" 20:24:54 DEBUG "pod": "ds-cts-0" 20:24:54 DEBUG "resultType": "vector", 20:24:54 DEBUG }, 20:24:54 DEBUG { 20:24:54 DEBUG [...] 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG "result": [ 20:24:54 DEBUG "metric": { 20:24:54 DEBUG [...] 20:24:54 DEBUG 20:24:54 DEBUG "instance": "10.142.0.43:9100" 20:24:54 DEBUG }, 20:24:54 DEBUG }, 20:24:54 DEBUG "result": [ 20:24:54 DEBUG "value": [ 20:24:54 DEBUG "metric": { 20:24:54 DEBUG 20:24:54 DEBUG "result": [ 20:24:54 DEBUG 20:24:54 DEBUG { 20:24:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG "value": [ 20:24:54 DEBUG { 20:24:54 DEBUG 20:24:54 DEBUG "pod": "ds-idrepo-0" 20:24:54 DEBUG 20:24:54 DEBUG { 20:24:54 DEBUG 20:24:54 DEBUG "metric": { 20:24:54 DEBUG }, 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "metric": { 20:24:54 DEBUG 20:24:54 DEBUG }, 20:24:54 DEBUG 20:24:54 DEBUG "metric": { 20:24:54 DEBUG 20:24:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:24:54 DEBUG "value": [ 20:24:54 DEBUG [...] 20:24:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:24:54 DEBUG 20:24:54 DEBUG "value": [ 20:24:54 DEBUG "pod": "ds-idrepo-2" 20:24:54 DEBUG "pod": "ds-cts-0" 20:24:54 DEBUG [...] 20:24:54 DEBUG }, 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG }, 20:24:54 DEBUG "71278504960" 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG }, 20:24:54 DEBUG [...] 20:24:54 DEBUG 20:24:54 DEBUG }, 20:24:54 DEBUG }, 20:24:54 DEBUG 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "value": [ 20:24:54 DEBUG ] 20:24:54 DEBUG [...] 20:24:54 DEBUG [...] 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "value": [ 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG } 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG "pod": "ds-cts-2" 20:24:54 DEBUG "6.724144598079293" 20:24:54 DEBUG ] 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG "0" 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "instance": "10.142.0.43:9100" 20:24:54 DEBUG 20:24:54 DEBUG }, 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG 20:24:54 DEBUG "pod": "ds-cts-2" 20:24:54 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG }, 20:24:54 DEBUG 20:24:54 DEBUG "value": [ 20:24:54 DEBUG } 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG }, 20:24:54 DEBUG }, 20:24:54 DEBUG ] 20:24:54 DEBUG 20:24:54 DEBUG [...] 20:24:54 DEBUG 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG ] 20:24:54 DEBUG "pod": "ds-idrepo-2" 20:24:54 DEBUG [...] 20:24:54 DEBUG "value": [ 20:24:54 DEBUG "value": [ 20:24:54 DEBUG } 20:24:54 DEBUG "pod": "ds-idrepo-2" 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:24:54 DEBUG "0" 20:24:54 DEBUG } 20:24:54 DEBUG }, 20:24:54 DEBUG 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG ] 20:24:54 DEBUG }, 20:24:54 DEBUG 20:24:54 DEBUG 20:24:54 DEBUG "0.03333333333333348" 20:24:54 DEBUG ] 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 20:24:54 DEBUG "0" 20:24:54 DEBUG "0" 20:24:54 DEBUG } 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG "0" 20:24:54 DEBUG "pod": "ds-cts-2" 20:24:54 DEBUG }, 20:24:54 DEBUG "value": [ 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG "0" 20:24:54 DEBUG ] 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG 20:24:54 DEBUG }, 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:24:54 DEBUG "value": [ 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG ] 20:24:54 DEBUG "0" 20:24:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG ] 20:24:54 DEBUG }, 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG }, 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG "value": [ 20:24:54 DEBUG "1814.6333333333332" 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG "value": [ 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG 1713381894, 20:24:54 DEBUG } 20:24:54 DEBUG "312673" 20:24:54 DEBUG ] 20:24:54 DEBUG ] 20:24:54 DEBUG "0" 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG ] 20:24:54 DEBUG ] 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 DEBUG } 20:24:54 INFO [loop_until]: OK (rc = 0) 20:24:54 DEBUG --- stdout --- 20:24:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 5499Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4418m 14% 5935Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9413m 31% 9573Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 78m 0% 2990Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 5545m 18% 9809Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 5011Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9616m 32% 9755Mi 8% gke-xlou-cdm-ds-0f601751-cf7b 27651m 92% 2913Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9497m 31% 8969Mi 8% gke-xlou-cdm-ds-0f601751-j488 4912m 16% 4765Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 5114m 17% 10926Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5201m 17% 5354Mi 4% gke-xlou-cdm-ds-0f601751-nmxm 5527m 18% 9422Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6865m 43% 4620Mi 7% 20:24:54 DEBUG --- stderr --- 20:24:54 DEBUG 20:25:54 INFO 20:25:54 INFO [loop_until]: kubectl --namespace=xlou top pods 20:25:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713381954" 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713381954" 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381954" 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 ERROR [http_cmd]: ERROR 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 ERROR --- status code --- 20:25:54 DEBUG --- status code --- 20:25:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 ERROR --- http response --- 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:25:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG { 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "instance": "10.142.0.107:9100" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "instance": "10.142.0.43:9100" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "0.06600000000000013" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "956858368" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "0.002" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [loop_until]: OK (rc = 0) 20:25:54 DEBUG --- stdout --- 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11843m 7250Mi am-b59c765b7-bgsxp 10735m 7267Mi am-b59c765b7-ch5q4 11855m 7057Mi ds-cts-0 11927m 7615Mi ds-cts-1 11182m 7504Mi ds-cts-2 12622m 7778Mi ds-idrepo-0 5198m 3076Mi ds-idrepo-1 5424m 2447Mi ds-idrepo-2 5270m 2430Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 965Mi idm-b85f9d978-wnhxb 11m 1010Mi lodemon-db4967764-9zt9t 8m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6956m 820Mi 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG --- stderr --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "1822.6" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "1839.7666666666667" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381954" 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "0" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "instance": "10.142.0.107:9100" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "instance": "10.142.0.43:9100" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "60930183.1" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "1839.9333333333334" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "pod": "ds-cts-0" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "pod": "ds-cts-2" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "0.000033333333333333335" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381954" 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "instance": "10.142.0.107:9100" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "instance": "10.142.0.43:9100" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "60925502.06666666" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381954" 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "pod": "ds-idrepo-0" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "pod": "ds-cts-2" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "0" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "0" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "instance": "10.142.0.107:9100" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "instance": "10.142.0.43:9100" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "0.041666666666666706" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "pod": "ds-cts-0" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "pod": "ds-cts-2" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "0" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "data": { 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "328055" 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713381954" 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713381954" 20:25:54 DEBUG --- status code --- 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG ----- output ----- 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:25:54 DEBUG { 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 DEBUG { 20:25:54 DEBUG ----- output ----- 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713381954" 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG { 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG "data": { 20:25:54 DEBUG "data": { 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG "data": { 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG "result": [ 20:25:54 DEBUG "result": [ 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG { 20:25:54 DEBUG { 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG "result": [ 20:25:54 DEBUG "metric": { 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG "metric": { 20:25:54 DEBUG { 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:25:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:25:54 DEBUG "metric": { 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG }, 20:25:54 DEBUG { 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG }, 20:25:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:25:54 DEBUG { 20:25:54 DEBUG "value": [ 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG "value": [ 20:25:54 DEBUG }, 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG 20:25:54 DEBUG "data": { 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:25:54 DEBUG 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:25:54 DEBUG "value": [ 20:25:54 DEBUG "data": { 20:25:54 DEBUG 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG 20:25:54 DEBUG "result": [ 20:25:54 DEBUG { 20:25:54 DEBUG 20:25:54 DEBUG { 20:25:54 DEBUG 20:25:54 DEBUG "result": [ 20:25:54 DEBUG [...] 20:25:54 DEBUG { 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG [...] 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG 20:25:54 DEBUG { 20:25:54 DEBUG 20:25:54 DEBUG "metric": { 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG "data": { 20:25:54 DEBUG 20:25:54 DEBUG "data": { 20:25:54 DEBUG [...] 20:25:54 DEBUG "metric": { 20:25:54 DEBUG 20:25:54 DEBUG "instance": "10.142.0.107:9100" 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG 20:25:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:25:54 DEBUG 20:25:54 DEBUG }, 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG "result": [ 20:25:54 DEBUG 20:25:54 DEBUG "result": [ 20:25:54 DEBUG 20:25:54 DEBUG }, 20:25:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:25:54 DEBUG "value": [ 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG { 20:25:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:25:54 DEBUG { 20:25:54 DEBUG 20:25:54 DEBUG "value": [ 20:25:54 DEBUG }, 20:25:54 DEBUG 20:25:54 DEBUG "metric": { 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:25:54 DEBUG }, 20:25:54 DEBUG "metric": { 20:25:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:25:54 DEBUG 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG "pod": "ds-cts-0" 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG "value": [ 20:25:54 DEBUG "pod": "ds-idrepo-0" 20:25:54 DEBUG }, 20:25:54 DEBUG 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG "0" 20:25:54 DEBUG [...] 20:25:54 DEBUG { 20:25:54 DEBUG 20:25:54 DEBUG "0" 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG [...] 20:25:54 DEBUG ] 20:25:54 DEBUG 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG 20:25:54 DEBUG ] 20:25:54 DEBUG 20:25:54 DEBUG "0" 20:25:54 DEBUG 20:25:54 DEBUG } 20:25:54 DEBUG 20:25:54 DEBUG "data": { 20:25:54 DEBUG 20:25:54 DEBUG } 20:25:54 DEBUG 20:25:54 DEBUG ] 20:25:54 DEBUG 20:25:54 DEBUG ] 20:25:54 DEBUG 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG [...] 20:25:54 DEBUG ] 20:25:54 DEBUG 20:25:54 DEBUG [...] 20:25:54 DEBUG 20:25:54 DEBUG } 20:25:54 DEBUG "instance": "10.142.0.43:9100" 20:25:54 DEBUG "result": [ 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG --- status code --- 20:25:54 INFO [http_cmd]: http status code OK 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 DEBUG 20:25:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:25:54 DEBUG } 20:25:54 DEBUG }, 20:25:54 DEBUG { 20:25:54 DEBUG 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- status code --- 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG "metric": { 20:25:54 DEBUG 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:25:54 DEBUG --- http response --- 20:25:54 DEBUG 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "pod": "ds-idrepo-0" 20:25:54 DEBUG 20:25:54 DEBUG } 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:25:54 DEBUG "pod": "ds-idrepo-2" 20:25:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "73678232576" 20:25:54 DEBUG }, 20:25:54 DEBUG "pod": "ds-cts-2" 20:25:54 DEBUG } 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG }, 20:25:54 DEBUG ----- output ----- 20:25:54 DEBUG "6.808289864052107" 20:25:54 DEBUG ] 20:25:54 DEBUG "value": [ 20:25:54 DEBUG }, 20:25:54 DEBUG { 20:25:54 DEBUG "value": [ 20:25:54 DEBUG { 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG 20:25:54 DEBUG "value": [ 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "status": "success", 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "data": { 20:25:54 DEBUG "0" 20:25:54 DEBUG "data": { 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG 20:25:54 DEBUG "0" 20:25:54 INFO 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG ] 20:25:54 DEBUG "resultType": "vector", 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 DEBUG [...] 20:25:54 DEBUG ] 20:25:54 INFO [loop_until]: kubectl --namespace=xlou top node 20:25:54 DEBUG "result": [ 20:25:54 DEBUG } 20:25:54 DEBUG "result": [ 20:25:54 DEBUG } 20:25:54 DEBUG 20:25:54 DEBUG } 20:25:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:25:54 DEBUG { 20:25:54 DEBUG ] 20:25:54 DEBUG { 20:25:54 DEBUG 20:25:54 DEBUG ] 20:25:54 DEBUG "metric": { 20:25:54 DEBUG } 20:25:54 DEBUG "metric": { 20:25:54 DEBUG 20:25:54 DEBUG } 20:25:54 DEBUG "pod": "ds-idrepo-0" 20:25:54 DEBUG } 20:25:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:25:54 DEBUG "pod": "ds-idrepo-2" 20:25:54 DEBUG } 20:25:54 DEBUG }, 20:25:54 DEBUG }, 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG "value": [ 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG "0" 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG ] 20:25:54 DEBUG [...] 20:25:54 DEBUG [...] 20:25:54 DEBUG } 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG ] 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG } 20:25:54 DEBUG 20:25:54 DEBUG 20:25:54 DEBUG } 20:25:54 DEBUG "pod": "ds-idrepo-2" 20:25:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:25:54 DEBUG }, 20:25:54 DEBUG }, 20:25:54 DEBUG "value": [ 20:25:54 DEBUG "value": [ 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG 1713381954, 20:25:54 DEBUG "0" 20:25:54 DEBUG "1839.9666666666667" 20:25:54 DEBUG ] 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 DEBUG ] 20:25:54 DEBUG ] 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 DEBUG } 20:25:54 INFO [loop_until]: OK (rc = 0) 20:25:54 DEBUG --- stdout --- 20:25:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 5510Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5224m 17% 5942Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11792m 39% 9634Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 2983Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11748m 39% 9763Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 163m 0% 5002Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11634m 38% 9858Mi 8% gke-xlou-cdm-ds-0f601751-cf7b 28172m 94% 2908Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11839m 39% 9094Mi 8% gke-xlou-cdm-ds-0f601751-j488 5584m 18% 4955Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 10736m 35% 10884Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5943m 19% 5485Mi 4% gke-xlou-cdm-ds-0f601751-nmxm 11924m 39% 9375Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6962m 43% 4622Mi 7% 20:25:54 DEBUG --- stderr --- 20:25:54 DEBUG 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "0.002" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382014" 20:26:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382014" 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382014" 20:26:54 ERROR [http_cmd]: ERROR 20:26:54 ERROR --- status code --- 20:26:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:26:54 ERROR --- http response --- 20:26:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:26:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "961593344" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "instance": "10.142.0.107:9100" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "instance": "10.142.0.43:9100" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "0.1713000000000003" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO 20:26:54 INFO [loop_until]: kubectl --namespace=xlou top pods 20:26:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "1821.8999999999999" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "1829.9" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382014" 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "instance": "10.142.0.107:9100" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "instance": "10.142.0.43:9100" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "61022313.46666667" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [loop_until]: OK (rc = 0) 20:26:54 DEBUG --- stdout --- 20:26:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12006m 7256Mi am-b59c765b7-bgsxp 11039m 7271Mi am-b59c765b7-ch5q4 12096m 7061Mi ds-cts-0 11676m 7607Mi ds-cts-1 11347m 7506Mi ds-cts-2 12817m 7759Mi ds-idrepo-0 5365m 3082Mi ds-idrepo-1 5365m 2517Mi ds-idrepo-2 5438m 2435Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 5m 1003Mi lodemon-db4967764-9zt9t 8m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6794m 822Mi 20:26:54 DEBUG --- stderr --- 20:26:54 DEBUG 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "0" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "1829.9333333333334" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "ds-cts-0" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "ds-cts-2" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "0" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382014" 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "instance": "10.142.0.107:9100" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "instance": "10.142.0.43:9100" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "61016680.6" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 DEBUG "status": "success", 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "0" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382014" 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "328547" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "0" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "ds-idrepo-0" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "ds-cts-2" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "0" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "instance": "10.142.0.107:9100" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "instance": "10.142.0.43:9100" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "0.09299999999999989" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "ds-cts-0" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "ds-cts-2" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "0" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "1829.3999999999999" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "0" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "0.00004587495375508262" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382014" 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382014" 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382014" 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "instance": "10.142.0.107:9100" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "instance": "10.142.0.43:9100" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "76122799104" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "6.801720297618757" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG "data": { 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG "result": [ 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:26:54 DEBUG { 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG "metric": { 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 INFO [http_cmd]: http status code OK 20:26:54 DEBUG "pod": "ds-idrepo-0" 20:26:54 DEBUG { 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG --- status code --- 20:26:54 DEBUG }, 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "data": { 20:26:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:26:54 DEBUG "value": [ 20:26:54 DEBUG "data": { 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG --- http response --- 20:26:54 DEBUG 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "result": [ 20:26:54 DEBUG 20:26:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:26:54 DEBUG "result": [ 20:26:54 DEBUG { 20:26:54 DEBUG 20:26:54 DEBUG ----- output ----- 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG [...] 20:26:54 DEBUG { 20:26:54 DEBUG "metric": { 20:26:54 DEBUG "pod": "ds-cts-0" 20:26:54 DEBUG 20:26:54 DEBUG "status": "success", 20:26:54 DEBUG "pod": "ds-idrepo-0" 20:26:54 DEBUG }, 20:26:54 DEBUG 20:26:54 DEBUG "data": { 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG "resultType": "vector", 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG "pod": "ds-idrepo-2" 20:26:54 DEBUG "result": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG }, 20:26:54 DEBUG { 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "value": [ 20:26:54 DEBUG "metric": { 20:26:54 DEBUG 20:26:54 DEBUG [...] 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "pod": "ds-idrepo-0" 20:26:54 DEBUG [...] 20:26:54 DEBUG 20:26:54 DEBUG "0" 20:26:54 DEBUG }, 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG ] 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG } 20:26:54 DEBUG 20:26:54 DEBUG 20:26:54 DEBUG "pod": "ds-cts-2" 20:26:54 DEBUG ] 20:26:54 DEBUG 20:26:54 DEBUG "pod": "ds-idrepo-2" 20:26:54 DEBUG }, 20:26:54 DEBUG } 20:26:54 DEBUG 20:26:54 DEBUG }, 20:26:54 DEBUG "value": [ 20:26:54 DEBUG } 20:26:54 DEBUG [...] 20:26:54 DEBUG "value": [ 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG "0" 20:26:54 DEBUG 20:26:54 DEBUG "0" 20:26:54 DEBUG ] 20:26:54 DEBUG 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG "pod": "ds-idrepo-2" 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG }, 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG "value": [ 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 DEBUG 1713382014, 20:26:54 DEBUG } 20:26:54 DEBUG "0" 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG ] 20:26:54 DEBUG } 20:26:54 DEBUG } 20:26:54 INFO 20:26:54 INFO [loop_until]: kubectl --namespace=xlou top node 20:26:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:26:54 INFO [loop_until]: OK (rc = 0) 20:26:54 DEBUG --- stdout --- 20:26:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 5505Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5235m 17% 5969Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12024m 40% 9636Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 2979Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12033m 40% 9753Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 166m 0% 5003Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12159m 40% 9866Mi 8% gke-xlou-cdm-ds-0f601751-cf7b 28115m 94% 2920Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11750m 39% 9103Mi 8% gke-xlou-cdm-ds-0f601751-j488 5617m 18% 4956Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11167m 37% 10873Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5897m 19% 5542Mi 4% gke-xlou-cdm-ds-0f601751-nmxm 12637m 42% 9360Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6946m 43% 4624Mi 7% 20:26:54 DEBUG --- stderr --- 20:26:54 DEBUG 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382074" 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "0.0025" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382074" 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "instance": "10.142.0.107:9100" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "instance": "10.142.0.43:9100" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "0.0951333333333333" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:27:54 ERROR [http_cmd]: ERROR 20:27:54 ERROR --- status code --- 20:27:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:27:54 ERROR --- http response --- 20:27:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:27:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382074" 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "1814.5666666666666" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "962637824" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382074" 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "1812.1" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "instance": "10.142.0.107:9100" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "instance": "10.142.0.43:9100" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "60710961.8" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO 20:27:54 INFO [loop_until]: kubectl --namespace=xlou top pods 20:27:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG { 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "1812.1666666666667" 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:27:54 DEBUG ] 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG } 20:27:54 DEBUG { 20:27:54 DEBUG ] 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG } 20:27:54 DEBUG "data": { 20:27:54 DEBUG } 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "0" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [loop_until]: OK (rc = 0) 20:27:54 DEBUG --- stdout --- 20:27:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11790m 8861Mi am-b59c765b7-bgsxp 10905m 7283Mi am-b59c765b7-ch5q4 11566m 8819Mi ds-cts-0 12445m 7604Mi ds-cts-1 11177m 7486Mi ds-cts-2 12798m 7766Mi ds-idrepo-0 5311m 3085Mi ds-idrepo-1 5481m 2562Mi ds-idrepo-2 5320m 2543Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 6m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 1m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6779m 824Mi 20:27:54 DEBUG --- stderr --- 20:27:54 DEBUG 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "ds-cts-0" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "ds-cts-2" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "0.0000666666666666667" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382074" 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "1812" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "ds-idrepo-0" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "ds-cts-2" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "0" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "329083" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "instance": "10.142.0.107:9100" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "instance": "10.142.0.43:9100" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "60705424.166666664" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "ds-idrepo-1" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "0" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382074" 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382074" 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "ds-idrepo-1" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "0" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "instance": "10.142.0.107:9100" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "instance": "10.142.0.43:9100" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "78722520064" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382074" 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "resultType": "vector", 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG "result": [ 20:27:54 DEBUG "result": [ 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG { 20:27:54 DEBUG { 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG "pod": "ds-cts-0" 20:27:54 DEBUG "instance": "10.142.0.107:9100" 20:27:54 DEBUG }, 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG { 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "data": { 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG "result": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG { 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "metric": { 20:27:54 DEBUG 20:27:54 DEBUG "pod": "ds-cts-2" 20:27:54 DEBUG "pod": "ds-idrepo-1" 20:27:54 DEBUG "instance": "10.142.0.43:9100" 20:27:54 DEBUG }, 20:27:54 DEBUG }, 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG "value": [ 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "0" 20:27:54 DEBUG 20:27:54 DEBUG "0.05500000000000007" 20:27:54 DEBUG ] 20:27:54 DEBUG 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG [...] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 DEBUG 20:27:54 DEBUG } 20:27:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "0" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "ds-idrepo-1" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "0" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "6.785217851272563" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382074" 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "data": { 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "ds-idrepo-0" 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "ds-idrepo-0" 20:27:54 DEBUG }, 20:27:54 DEBUG "pod": "ds-cts-0" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "ds-idrepo-2" 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG }, 20:27:54 DEBUG "pod": "ds-idrepo-2" 20:27:54 DEBUG 20:27:54 DEBUG "value": [ 20:27:54 INFO [http_cmd]: http status code OK 20:27:54 DEBUG }, 20:27:54 DEBUG "pod": "ds-cts-2" 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG --- status code --- 20:27:54 DEBUG "value": [ 20:27:54 DEBUG }, 20:27:54 DEBUG "0" 20:27:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG ] 20:27:54 DEBUG --- http response --- 20:27:54 DEBUG "0" 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:27:54 DEBUG "0" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ----- output ----- 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG { 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 DEBUG "status": "success", 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG "data": { 20:27:54 DEBUG } 20:27:54 DEBUG "resultType": "vector", 20:27:54 DEBUG } 20:27:54 DEBUG "result": [ 20:27:54 DEBUG { 20:27:54 DEBUG "metric": { 20:27:54 DEBUG "pod": "ds-idrepo-0" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG [...] 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG 20:27:54 DEBUG "pod": "ds-idrepo-2" 20:27:54 DEBUG }, 20:27:54 DEBUG "value": [ 20:27:54 DEBUG 1713382074, 20:27:54 DEBUG "0" 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG ] 20:27:54 DEBUG } 20:27:54 DEBUG } 20:27:54 INFO 20:27:54 INFO [loop_until]: kubectl --namespace=xlou top node 20:27:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:27:54 INFO [loop_until]: OK (rc = 0) 20:27:54 DEBUG --- stdout --- 20:27:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 5507Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5208m 17% 5969Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11384m 38% 11378Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 2977Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12265m 41% 9767Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 5006Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11902m 39% 11466Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28247m 94% 2914Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 12022m 40% 9124Mi 8% gke-xlou-cdm-ds-0f601751-j488 5553m 18% 5059Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11218m 37% 10880Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5871m 19% 5599Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12845m 43% 9370Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6718m 42% 4625Mi 7% 20:27:54 DEBUG --- stderr --- 20:27:54 DEBUG 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "0.003" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382134" 20:28:54 ERROR [http_cmd]: ERROR 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 ERROR --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 ERROR --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:28:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:28:54 DEBUG ----- output ----- 20:28:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG "instance": "10.142.0.107:9100" 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG }, 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG "value": [ 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:28:54 DEBUG [...] 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG 20:28:54 DEBUG { 20:28:54 DEBUG 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG 20:28:54 DEBUG "data": { 20:28:54 DEBUG "instance": "10.142.0.43:9100" 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG }, 20:28:54 DEBUG "result": [ 20:28:54 DEBUG "value": [ 20:28:54 DEBUG { 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "0.1745" 20:28:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:28:54 DEBUG ] 20:28:54 DEBUG }, 20:28:54 DEBUG } 20:28:54 DEBUG "value": [ 20:28:54 DEBUG ] 20:28:54 DEBUG 20:28:54 DEBUG } 20:28:54 DEBUG 20:28:54 DEBUG } 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "1839.9666666666667" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "961376256" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382134" 20:28:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "1831.7666666666667" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "instance": "10.142.0.107:9100" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "instance": "10.142.0.43:9100" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "60750750.599999994" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "1831.3333333333333" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "0" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO 20:28:54 INFO [loop_until]: kubectl --namespace=xlou top pods 20:28:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "pod": "ds-idrepo-0" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "pod": "ds-cts-2" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "0" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "pod": "ds-cts-0" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "pod": "ds-cts-2" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "0" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382134" 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 INFO [loop_until]: OK (rc = 0) 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- stdout --- 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11794m 8891Mi am-b59c765b7-bgsxp 10833m 8820Mi am-b59c765b7-ch5q4 11619m 8847Mi ds-cts-0 12452m 7606Mi ds-cts-1 11346m 7493Mi ds-cts-2 12797m 7784Mi ds-idrepo-0 5307m 3087Mi ds-idrepo-1 5355m 2677Mi ds-idrepo-2 5270m 2547Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6689m 826Mi 20:28:54 DEBUG --- stderr --- 20:28:54 DEBUG 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG { 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG "data": { 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:28:54 DEBUG { 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG "metric": { 20:28:54 DEBUG { 20:28:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG }, 20:28:54 DEBUG "data": { 20:28:54 DEBUG "value": [ 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG 20:28:54 DEBUG "result": [ 20:28:54 DEBUG 20:28:54 DEBUG { 20:28:54 DEBUG 20:28:54 DEBUG "metric": { 20:28:54 DEBUG [...] 20:28:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:28:54 DEBUG 20:28:54 DEBUG }, 20:28:54 DEBUG 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:28:54 DEBUG 20:28:54 DEBUG }, 20:28:54 DEBUG 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 DEBUG "value": [ 20:28:54 DEBUG [...] 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG 20:28:54 DEBUG "1831.7" 20:28:54 DEBUG 20:28:54 DEBUG ] 20:28:54 DEBUG 20:28:54 DEBUG } 20:28:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:28:54 DEBUG ] 20:28:54 DEBUG }, 20:28:54 DEBUG } 20:28:54 DEBUG "value": [ 20:28:54 DEBUG } 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "328480" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "instance": "10.142.0.107:9100" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "instance": "10.142.0.43:9100" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "60745128.233333334" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "3606.0384158578045" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382134" 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "0" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "instance": "10.142.0.107:9100" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "instance": "10.142.0.43:9100" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "81432460288" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG { 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG "data": { 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG { 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "metric": { 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG "data": { 20:28:54 DEBUG "pod": "ds-cts-0" 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG { 20:28:54 DEBUG "resultType": "vector", 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG }, 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG "value": [ 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG 20:28:54 DEBUG { 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:28:54 DEBUG "metric": { 20:28:54 DEBUG 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:28:54 DEBUG "metric": { 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:28:54 DEBUG [...] 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG "instance": "10.142.0.107:9100" 20:28:54 DEBUG { 20:28:54 DEBUG }, 20:28:54 DEBUG 20:28:54 DEBUG { 20:28:54 DEBUG }, 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "value": [ 20:28:54 DEBUG "data": { 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "data": { 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG 20:28:54 DEBUG "pod": "ds-cts-2" 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG 20:28:54 DEBUG "result": [ 20:28:54 DEBUG 20:28:54 DEBUG }, 20:28:54 DEBUG "result": [ 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG 20:28:54 DEBUG { 20:28:54 DEBUG [...] 20:28:54 DEBUG "value": [ 20:28:54 DEBUG { 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [...] 20:28:54 DEBUG "metric": { 20:28:54 DEBUG 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "metric": { 20:28:54 DEBUG 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:28:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:28:54 DEBUG 20:28:54 DEBUG "0" 20:28:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:28:54 DEBUG 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG }, 20:28:54 DEBUG 20:28:54 DEBUG ] 20:28:54 DEBUG }, 20:28:54 DEBUG 20:28:54 DEBUG { 20:28:54 DEBUG "value": [ 20:28:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:28:54 DEBUG } 20:28:54 DEBUG "value": [ 20:28:54 DEBUG "instance": "10.142.0.43:9100" 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG 20:28:54 DEBUG }, 20:28:54 DEBUG ] 20:28:54 DEBUG 20:28:54 DEBUG }, 20:28:54 DEBUG "data": { 20:28:54 DEBUG 20:28:54 DEBUG "value": [ 20:28:54 DEBUG } 20:28:54 DEBUG 20:28:54 DEBUG "value": [ 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG } 20:28:54 DEBUG 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "result": [ 20:28:54 DEBUG [...] 20:28:54 DEBUG "0" 20:28:54 DEBUG [...] 20:28:54 DEBUG "0.08766666666666667" 20:28:54 DEBUG { 20:28:54 DEBUG 20:28:54 DEBUG ] 20:28:54 DEBUG 20:28:54 DEBUG ] 20:28:54 DEBUG "metric": { 20:28:54 DEBUG 20:28:54 DEBUG } 20:28:54 DEBUG 20:28:54 DEBUG } 20:28:54 DEBUG "pod": "ds-cts-0" 20:28:54 DEBUG 20:28:54 DEBUG ] 20:28:54 DEBUG 20:28:54 DEBUG ] 20:28:54 DEBUG }, 20:28:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:28:54 DEBUG } 20:28:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:28:54 DEBUG } 20:28:54 DEBUG "value": [ 20:28:54 DEBUG }, 20:28:54 DEBUG } 20:28:54 DEBUG }, 20:28:54 DEBUG } 20:28:54 DEBUG 20:28:54 DEBUG "value": [ 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG 20:28:54 DEBUG "4.8536658327562066" 20:28:54 DEBUG "6.772597024038192" 20:28:54 DEBUG [...] 20:28:54 DEBUG ] 20:28:54 DEBUG ] 20:28:54 DEBUG 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 DEBUG 20:28:54 DEBUG ] 20:28:54 DEBUG ] 20:28:54 DEBUG 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 DEBUG "pod": "ds-cts-2" 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "0" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382134" 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "pod": "ds-idrepo-0" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "pod": "ds-idrepo-2" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "0" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "pod": "ds-idrepo-0" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "pod": "ds-idrepo-2" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "0" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO [http_cmd]: http status code OK 20:28:54 DEBUG --- status code --- 20:28:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:28:54 DEBUG --- http response --- 20:28:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:28:54 DEBUG ----- output ----- 20:28:54 DEBUG { 20:28:54 DEBUG "status": "success", 20:28:54 DEBUG "data": { 20:28:54 DEBUG "resultType": "vector", 20:28:54 DEBUG "result": [ 20:28:54 DEBUG { 20:28:54 DEBUG "metric": { 20:28:54 DEBUG "pod": "ds-idrepo-0" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG [...] 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG 20:28:54 DEBUG "pod": "ds-idrepo-2" 20:28:54 DEBUG }, 20:28:54 DEBUG "value": [ 20:28:54 DEBUG 1713382134, 20:28:54 DEBUG "0" 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG ] 20:28:54 DEBUG } 20:28:54 DEBUG } 20:28:54 INFO 20:28:54 INFO [loop_until]: kubectl --namespace=xlou top node 20:28:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:28:54 INFO [loop_until]: OK (rc = 0) 20:28:54 DEBUG --- stdout --- 20:28:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 5508Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5223m 17% 5972Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11749m 39% 11420Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 2979Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12467m 41% 9780Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 4995Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11934m 39% 11493Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28018m 93% 2923Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11662m 39% 10652Mi 9% gke-xlou-cdm-ds-0f601751-j488 5542m 18% 5068Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11537m 38% 10892Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5857m 19% 5668Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12790m 42% 9385Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6928m 43% 4624Mi 7% 20:28:54 DEBUG --- stderr --- 20:28:54 DEBUG 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382194" 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382194" 20:29:54 ERROR [http_cmd]: ERROR 20:29:54 ERROR --- status code --- 20:29:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:29:54 ERROR --- http response --- 20:29:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:29:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "0.003" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "965111808" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382194" 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG ----- output ----- 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG "data": { 20:29:54 DEBUG { 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG "result": [ 20:29:54 DEBUG "data": { 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "metric": { 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:29:54 DEBUG "result": [ 20:29:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG }, 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "value": [ 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "instance": "10.142.0.107:9100" 20:29:54 DEBUG 20:29:54 DEBUG "data": { 20:29:54 DEBUG }, 20:29:54 DEBUG 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG "result": [ 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG { 20:29:54 DEBUG 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382194" 20:29:54 DEBUG 20:29:54 DEBUG "metric": { 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG }, 20:29:54 DEBUG 20:29:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG }, 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG "instance": "10.142.0.43:9100" 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG 20:29:54 DEBUG }, 20:29:54 DEBUG "1829.2666666666667" 20:29:54 DEBUG [...] 20:29:54 DEBUG "value": [ 20:29:54 DEBUG ] 20:29:54 DEBUG 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG } 20:29:54 DEBUG 20:29:54 DEBUG "0.09826666666666596" 20:29:54 DEBUG ] 20:29:54 DEBUG 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 DEBUG }, 20:29:54 DEBUG ] 20:29:54 DEBUG "value": [ 20:29:54 DEBUG } 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG } 20:29:54 DEBUG "1818.4333333333334" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "instance": "10.142.0.107:9100" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "instance": "10.142.0.43:9100" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "60946438.43333334" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "1818.4" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "pod": "ds-idrepo-0" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "ds-cts-2" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "0" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "0" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382194" 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG { 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "pod": "ds-cts-0" 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:29:54 DEBUG }, 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG "value": [ 20:29:54 DEBUG { 20:29:54 DEBUG { 20:29:54 DEBUG 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG 20:29:54 DEBUG "data": { 20:29:54 DEBUG "data": { 20:29:54 DEBUG 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG [...] 20:29:54 DEBUG "result": [ 20:29:54 DEBUG "result": [ 20:29:54 DEBUG 20:29:54 DEBUG { 20:29:54 DEBUG { 20:29:54 DEBUG 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG 20:29:54 DEBUG "instance": "10.142.0.107:9100" 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:29:54 DEBUG "pod": "ds-cts-2" 20:29:54 DEBUG }, 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG }, 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG "value": [ 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:29:54 DEBUG "0.000017104444444444444" 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG ] 20:29:54 DEBUG [...] 20:29:54 DEBUG [...] 20:29:54 DEBUG { 20:29:54 DEBUG } 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG ] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "data": { 20:29:54 DEBUG } 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG } 20:29:54 DEBUG "instance": "10.142.0.43:9100" 20:29:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:29:54 DEBUG "result": [ 20:29:54 DEBUG }, 20:29:54 DEBUG }, 20:29:54 DEBUG { 20:29:54 DEBUG "value": [ 20:29:54 DEBUG "value": [ 20:29:54 DEBUG "metric": { 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:29:54 DEBUG "60940224.5" 20:29:54 DEBUG "1818.1" 20:29:54 DEBUG }, 20:29:54 DEBUG ] 20:29:54 DEBUG ] 20:29:54 DEBUG "value": [ 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 DEBUG 20:29:54 DEBUG ] 20:29:54 DEBUG ] 20:29:54 DEBUG 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 DEBUG 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "ds-cts-0" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "322728.4170526066" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382194" 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "instance": "10.142.0.107:9100" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "instance": "10.142.0.43:9100" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "84711003136" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [loop_until]: kubectl --namespace=xlou top pods 20:29:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382194" 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG [...] 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:29:54 DEBUG 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG 20:29:54 DEBUG { 20:29:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG }, 20:29:54 DEBUG "data": { 20:29:54 DEBUG "value": [ 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "result": [ 20:29:54 DEBUG "328386" 20:29:54 DEBUG { 20:29:54 DEBUG ] 20:29:54 DEBUG "metric": { 20:29:54 DEBUG } 20:29:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:29:54 DEBUG ] 20:29:54 DEBUG }, 20:29:54 DEBUG } 20:29:54 DEBUG "value": [ 20:29:54 DEBUG } 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "ds-cts-0" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "12.360281409571321" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "6.78295762180354" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "ds-cts-0" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "54649691.96057475" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382194" 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [loop_until]: OK (rc = 0) 20:29:54 DEBUG --- stdout --- 20:29:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11818m 8890Mi am-b59c765b7-bgsxp 10824m 8851Mi am-b59c765b7-ch5q4 11764m 8853Mi ds-cts-0 12391m 7615Mi ds-cts-1 11416m 7504Mi ds-cts-2 12834m 7788Mi ds-idrepo-0 5418m 3088Mi ds-idrepo-1 5396m 2679Mi ds-idrepo-2 5192m 2548Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 9m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6596m 829Mi 20:29:54 DEBUG --- stderr --- 20:29:54 DEBUG 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "pod": "ds-cts-0" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "ds-cts-2" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "0" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "instance": "10.142.0.107:9100" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "instance": "10.142.0.43:9100" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "0.05733333333333346" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "pod": "ds-cts-0" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "ds-cts-2" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "0" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "ds-cts-0" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "0" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "pod": "ds-idrepo-0" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "ds-idrepo-2" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "0" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382194" 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "pod": "ds-idrepo-0" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "ds-idrepo-2" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "0" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO [http_cmd]: http status code OK 20:29:54 DEBUG --- status code --- 20:29:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:29:54 DEBUG --- http response --- 20:29:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:29:54 DEBUG ----- output ----- 20:29:54 DEBUG { 20:29:54 DEBUG "status": "success", 20:29:54 DEBUG "data": { 20:29:54 DEBUG "resultType": "vector", 20:29:54 DEBUG "result": [ 20:29:54 DEBUG { 20:29:54 DEBUG "metric": { 20:29:54 DEBUG "pod": "ds-idrepo-0" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG [...] 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG 20:29:54 DEBUG "pod": "ds-idrepo-2" 20:29:54 DEBUG }, 20:29:54 DEBUG "value": [ 20:29:54 DEBUG 1713382194, 20:29:54 DEBUG "0" 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG ] 20:29:54 DEBUG } 20:29:54 DEBUG } 20:29:54 INFO 20:29:54 INFO [loop_until]: kubectl --namespace=xlou top node 20:29:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:29:54 INFO [loop_until]: OK (rc = 0) 20:29:54 DEBUG --- stdout --- 20:29:54 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 80m 0% 5507Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5264m 17% 5955Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11787m 39% 11431Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 2995Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12331m 41% 9785Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 170m 0% 5008Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11940m 39% 11501Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28132m 94% 2933Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11603m 38% 10691Mi 9% gke-xlou-cdm-ds-0f601751-j488 5556m 18% 5079Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11500m 38% 10897Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5608m 18% 5734Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12926m 43% 9389Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6933m 43% 4623Mi 7% 20:29:54 DEBUG --- stderr --- 20:29:54 DEBUG 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 ERROR [http_cmd]: ERROR 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 ERROR --- status code --- 20:30:54 DEBUG --- status code --- 20:30:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 ERROR --- http response --- 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:30:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:30:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382254" 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "0.003" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "969060352" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382254" 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "1812" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG ----- output ----- 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG { 20:30:54 DEBUG { 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG "data": { 20:30:54 DEBUG "data": { 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:54 DEBUG "result": [ 20:30:54 DEBUG "result": [ 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG { 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "instance": "10.142.0.107:9100" 20:30:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:30:54 DEBUG "data": { 20:30:54 DEBUG }, 20:30:54 DEBUG }, 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "value": [ 20:30:54 DEBUG "value": [ 20:30:54 DEBUG "result": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG { 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "metric": { 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:30:54 DEBUG [...] 20:30:54 DEBUG [...] 20:30:54 DEBUG }, 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "instance": "10.142.0.43:9100" 20:30:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:30:54 DEBUG 20:30:54 DEBUG }, 20:30:54 DEBUG }, 20:30:54 DEBUG [...] 20:30:54 DEBUG "value": [ 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG 20:30:54 DEBUG "0.17510000000000048" 20:30:54 DEBUG "1819.7333333333333" 20:30:54 DEBUG 20:30:54 DEBUG ] 20:30:54 DEBUG ] 20:30:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 DEBUG }, 20:30:54 DEBUG ] 20:30:54 DEBUG ] 20:30:54 DEBUG "value": [ 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 DEBUG "1819.4666666666667" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "instance": "10.142.0.107:9100" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "instance": "10.142.0.43:9100" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "60758506.4" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "0" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "pod": "ds-idrepo-0" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "ds-cts-2" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "0" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "0" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "0" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "pod": "ds-cts-0" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "ds-cts-2" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "0" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG { 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG "data": { 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:54 DEBUG "resultType": "vector", 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG "result": [ 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG { 20:30:54 DEBUG { 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "metric": { 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG "data": { 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG }, 20:30:54 DEBUG "result": [ 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "value": [ 20:30:54 DEBUG { 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG 20:30:54 DEBUG "instance": "10.142.0.107:9100" 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG 20:30:54 DEBUG }, 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG [...] 20:30:54 DEBUG "value": [ 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:30:54 DEBUG }, 20:30:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:30:54 DEBUG [...] 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG }, 20:30:54 DEBUG 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "1819.8999999999999" 20:30:54 DEBUG "instance": "10.142.0.43:9100" 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG ] 20:30:54 DEBUG }, 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG } 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG ] 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG } 20:30:54 DEBUG "60753014.36666666" 20:30:54 DEBUG 20:30:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:30:54 DEBUG }, 20:30:54 DEBUG } 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG ] 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG } 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "6.770449777180898" 20:30:54 DEBUG } 20:30:54 DEBUG "327850" 20:30:54 DEBUG ] 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "instance": "10.142.0.107:9100" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "instance": "10.142.0.43:9100" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "87502064640" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "0" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO 20:30:54 INFO [loop_until]: kubectl --namespace=xlou top pods 20:30:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "pod": "ds-cts-0" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "ds-cts-2" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "0" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "instance": "10.142.0.107:9100" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "instance": "10.142.0.43:9100" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "0.08700000000000015" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "pod": "ds-cts-0" 20:30:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:30:54 DEBUG }, 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "ds-cts-2" 20:30:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:30:54 DEBUG }, 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "0" 20:30:54 DEBUG "0" 20:30:54 DEBUG ] 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [loop_until]: OK (rc = 0) 20:30:54 DEBUG --- stdout --- 20:30:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11944m 8892Mi am-b59c765b7-bgsxp 10999m 8851Mi am-b59c765b7-ch5q4 11818m 8854Mi ds-cts-0 12231m 7631Mi ds-cts-1 11947m 7517Mi ds-cts-2 13198m 7772Mi ds-idrepo-0 5444m 3089Mi ds-idrepo-1 5263m 2681Mi ds-idrepo-2 5138m 2565Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6663m 831Mi 20:30:54 DEBUG --- stderr --- 20:30:54 DEBUG 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "pod": "ds-idrepo-0" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "ds-idrepo-2" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "0" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382254" 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "pod": "ds-idrepo-0" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "ds-idrepo-2" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "0" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO [http_cmd]: http status code OK 20:30:54 DEBUG --- status code --- 20:30:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:54 DEBUG --- http response --- 20:30:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:54 DEBUG ----- output ----- 20:30:54 DEBUG { 20:30:54 DEBUG "status": "success", 20:30:54 DEBUG "data": { 20:30:54 DEBUG "resultType": "vector", 20:30:54 DEBUG "result": [ 20:30:54 DEBUG { 20:30:54 DEBUG "metric": { 20:30:54 DEBUG "pod": "ds-idrepo-0" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG [...] 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG 20:30:54 DEBUG "pod": "ds-idrepo-2" 20:30:54 DEBUG }, 20:30:54 DEBUG "value": [ 20:30:54 DEBUG 1713382254, 20:30:54 DEBUG "0" 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG ] 20:30:54 DEBUG } 20:30:54 DEBUG } 20:30:54 INFO 20:30:54 INFO [loop_until]: kubectl --namespace=xlou top node 20:30:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:55 INFO [loop_until]: OK (rc = 0) 20:30:55 DEBUG --- stdout --- 20:30:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 5508Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5159m 17% 5968Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11803m 39% 11429Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 2989Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12218m 40% 9796Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5000Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12002m 40% 11487Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27981m 93% 2930Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11670m 39% 10695Mi 9% gke-xlou-cdm-ds-0f601751-j488 5511m 18% 5091Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11593m 38% 10908Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5787m 19% 5729Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13098m 43% 9395Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6917m 43% 4621Mi 7% 20:30:55 DEBUG --- stderr --- 20:30:55 DEBUG 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382314" 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 ERROR [http_cmd]: ERROR 20:31:54 ERROR --- status code --- 20:31:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:31:54 ERROR --- http response --- 20:31:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:31:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "0.003" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "970825728" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382314" 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "1828" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "1828.1333333333334" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "instance": "10.142.0.107:9100" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "instance": "10.142.0.43:9100" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "60769680.7" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "1831.7333333333333" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "instance": "10.142.0.107:9100" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "instance": "10.142.0.43:9100" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "0.15419999999999923" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG { 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "ds-idrepo-0" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "ds-cts-2" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "0" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "ds-cts-0" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "202823.15789473685" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "0" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "ds-cts-0" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "8.49191170650792" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382314" 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "ds-cts-0" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "ds-cts-2" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "0.00018771111111111108" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG { 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG "data": { 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG { 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "metric": { 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG "data": { 20:31:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:31:54 DEBUG { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG }, 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG "value": [ 20:31:54 DEBUG "data": { 20:31:54 DEBUG { 20:31:54 DEBUG 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "metric": { 20:31:54 DEBUG 20:31:54 DEBUG "result": [ 20:31:54 DEBUG "instance": "10.142.0.107:9100" 20:31:54 DEBUG 20:31:54 DEBUG { 20:31:54 DEBUG }, 20:31:54 DEBUG [...] 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG }, 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG }, 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG "1828.3666666666666" 20:31:54 DEBUG 20:31:54 DEBUG "instance": "10.142.0.43:9100" 20:31:54 DEBUG ] 20:31:54 DEBUG 20:31:54 DEBUG }, 20:31:54 DEBUG } 20:31:54 DEBUG 20:31:54 DEBUG "value": [ 20:31:54 DEBUG ] 20:31:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG } 20:31:54 DEBUG }, 20:31:54 DEBUG "60763834.2" 20:31:54 DEBUG } 20:31:54 DEBUG "value": [ 20:31:54 DEBUG ] 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG } 20:31:54 DEBUG "6.781759736424738" 20:31:54 DEBUG ] 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "327859.5" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "ds-cts-0" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "27575836.547368422" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "instance": "10.142.0.107:9100" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "instance": "10.142.0.43:9100" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "90869818368" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "ds-cts-0" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "ds-cts-2" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "0" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO 20:31:54 INFO [loop_until]: kubectl --namespace=xlou top pods 20:31:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "ds-cts-0" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "ds-cts-2" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "0" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "instance": "10.142.0.107:9100" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "instance": "10.142.0.43:9100" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "0.08633333333333316" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "ds-cts-0" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "25719556.86124402" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "ds-idrepo-0" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "ds-idrepo-2" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "0" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "ds-idrepo-0" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382314" 20:31:54 DEBUG "pod": "ds-idrepo-2" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "0" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:54 INFO [loop_until]: OK (rc = 0) 20:31:54 DEBUG --- stdout --- 20:31:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11866m 8893Mi am-b59c765b7-bgsxp 10754m 8713Mi am-b59c765b7-ch5q4 11992m 8853Mi ds-cts-0 12407m 7634Mi ds-cts-1 11526m 7505Mi ds-cts-2 13048m 7781Mi ds-idrepo-0 5232m 3091Mi ds-idrepo-1 5437m 2699Mi ds-idrepo-2 5378m 2574Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 8m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 9m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7057m 833Mi 20:31:54 DEBUG --- stderr --- 20:31:54 DEBUG 20:31:54 INFO [http_cmd]: http status code OK 20:31:54 DEBUG --- status code --- 20:31:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:54 DEBUG --- http response --- 20:31:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:54 DEBUG ----- output ----- 20:31:54 DEBUG { 20:31:54 DEBUG "status": "success", 20:31:54 DEBUG "data": { 20:31:54 DEBUG "resultType": "vector", 20:31:54 DEBUG "result": [ 20:31:54 DEBUG { 20:31:54 DEBUG "metric": { 20:31:54 DEBUG "pod": "ds-idrepo-0" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG [...] 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG 20:31:54 DEBUG "pod": "ds-idrepo-2" 20:31:54 DEBUG }, 20:31:54 DEBUG "value": [ 20:31:54 DEBUG 1713382314, 20:31:54 DEBUG "0" 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG ] 20:31:54 DEBUG } 20:31:54 DEBUG } 20:31:55 INFO 20:31:55 INFO [loop_until]: kubectl --namespace=xlou top node 20:31:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:31:55 INFO [loop_until]: OK (rc = 0) 20:31:55 DEBUG --- stdout --- 20:31:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 78m 0% 5498Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5139m 17% 5969Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11806m 39% 11429Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 2991Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12309m 41% 9793Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 145m 0% 4995Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12014m 40% 11487Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27992m 93% 2915Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11647m 39% 10553Mi 9% gke-xlou-cdm-ds-0f601751-j488 5526m 18% 5099Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11547m 38% 10903Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5789m 19% 5747Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12962m 43% 9406Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6935m 43% 4623Mi 7% 20:31:55 DEBUG --- stderr --- 20:31:55 DEBUG 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382374" 20:32:54 ERROR [http_cmd]: ERROR 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 ERROR --- status code --- 20:32:54 DEBUG --- status code --- 20:32:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 ERROR --- http response --- 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:32:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "0.003" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "972124160" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "1821.8666666666666" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "instance": "10.142.0.107:9100" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "instance": "10.142.0.43:9100" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "60885671.5" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "1818.2333333333333" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "1821.9666666666667" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382374" 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "instance": "10.142.0.107:9100" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "instance": "10.142.0.43:9100" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "0.1539999999999992" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "ds-idrepo-0" 20:32:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:32:54 DEBUG }, 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "ds-cts-2" 20:32:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:32:54 DEBUG }, 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "0" 20:32:54 DEBUG "64009.63095436679" 20:32:54 DEBUG ] 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "ds-cts-0" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "ds-cts-2" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "0.000017104444444444444" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "0" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "10.33233515670611" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG "result": [ 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG { 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "instance": "10.142.0.107:9100" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:54 DEBUG 20:32:54 DEBUG ----- output ----- 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG [...] 20:32:54 DEBUG { 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG 20:32:54 DEBUG "data": { 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "instance": "10.142.0.43:9100" 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:54 DEBUG "result": [ 20:32:54 DEBUG }, 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "value": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:32:54 DEBUG "60879782.233333334" 20:32:54 DEBUG "data": { 20:32:54 DEBUG }, 20:32:54 DEBUG ] 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "value": [ 20:32:54 DEBUG } 20:32:54 DEBUG "result": [ 20:32:54 DEBUG 20:32:54 DEBUG ] 20:32:54 DEBUG { 20:32:54 DEBUG 20:32:54 DEBUG } 20:32:54 DEBUG "metric": { 20:32:54 DEBUG 20:32:54 DEBUG } 20:32:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:32:54 DEBUG [...] 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382374" 20:32:54 DEBUG }, 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG 20:32:54 DEBUG "value": [ 20:32:54 DEBUG --- status code --- 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG 20:32:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:54 DEBUG [...] 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:32:54 DEBUG }, 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG "value": [ 20:32:54 DEBUG { 20:32:54 DEBUG 20:32:54 DEBUG { 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "6.787616714280064" 20:32:54 DEBUG "data": { 20:32:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:32:54 DEBUG "data": { 20:32:54 DEBUG ] 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG }, 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG } 20:32:54 DEBUG "result": [ 20:32:54 DEBUG "value": [ 20:32:54 DEBUG "result": [ 20:32:54 DEBUG ] 20:32:54 DEBUG { 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG { 20:32:54 DEBUG } 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "1821.1333333333332" 20:32:54 DEBUG "metric": { 20:32:54 DEBUG } 20:32:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:32:54 DEBUG ] 20:32:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:32:54 DEBUG }, 20:32:54 DEBUG } 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG ] 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG } 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG } 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:32:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:32:54 DEBUG }, 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "328174" 20:32:54 DEBUG "0" 20:32:54 DEBUG ] 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "instance": "10.142.0.107:9100" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "instance": "10.142.0.43:9100" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "94042571776" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "ds-cts-0" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "ds-cts-2" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "0" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "0" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382374" 20:32:54 INFO 20:32:54 INFO [loop_until]: kubectl --namespace=xlou top pods 20:32:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "ds-cts-0" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "ds-cts-2" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "0" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "instance": "10.142.0.107:9100" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "instance": "10.142.0.43:9100" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "0.07966666666666661" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "ds-idrepo-0" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "ds-idrepo-2" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "0" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "ds-idrepo-0" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "ds-idrepo-2" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "0" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382374" 20:32:54 INFO [http_cmd]: http status code OK 20:32:54 DEBUG --- status code --- 20:32:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:54 DEBUG --- http response --- 20:32:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:54 DEBUG ----- output ----- 20:32:54 DEBUG { 20:32:54 DEBUG "status": "success", 20:32:54 DEBUG "data": { 20:32:54 DEBUG "resultType": "vector", 20:32:54 DEBUG "result": [ 20:32:54 DEBUG { 20:32:54 DEBUG "metric": { 20:32:54 DEBUG "pod": "ds-idrepo-0" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG [...] 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG 20:32:54 DEBUG "pod": "ds-idrepo-2" 20:32:54 DEBUG }, 20:32:54 DEBUG "value": [ 20:32:54 DEBUG 1713382374, 20:32:54 DEBUG "0" 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG ] 20:32:54 DEBUG } 20:32:54 DEBUG } 20:32:54 INFO [loop_until]: OK (rc = 0) 20:32:54 DEBUG --- stdout --- 20:32:54 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11781m 8892Mi am-b59c765b7-bgsxp 10782m 8717Mi am-b59c765b7-ch5q4 11659m 8853Mi ds-cts-0 12647m 7632Mi ds-cts-1 11811m 7513Mi ds-cts-2 12737m 7803Mi ds-idrepo-0 5251m 3092Mi ds-idrepo-1 5256m 2699Mi ds-idrepo-2 5248m 2591Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 10m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6794m 835Mi 20:32:54 DEBUG --- stderr --- 20:32:54 DEBUG 20:32:55 INFO 20:32:55 INFO [loop_until]: kubectl --namespace=xlou top node 20:32:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:32:55 INFO [loop_until]: OK (rc = 0) 20:32:55 DEBUG --- stdout --- 20:32:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 5499Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4956m 16% 5979Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11852m 39% 11421Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 2989Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12533m 41% 9801Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 169m 0% 5003Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12060m 40% 11497Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28038m 93% 2913Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11429m 38% 10563Mi 9% gke-xlou-cdm-ds-0f601751-j488 5514m 18% 5120Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11576m 38% 10915Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5749m 19% 5741Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12782m 42% 9413Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6934m 43% 4626Mi 7% 20:32:55 DEBUG --- stderr --- 20:32:55 DEBUG 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382434" 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382434" 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 ERROR [http_cmd]: ERROR 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 ERROR --- status code --- 20:33:54 DEBUG --- status code --- 20:33:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 ERROR --- http response --- 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:33:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "0.003" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "974536704" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "1827.4333333333334" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382434" 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "1826.9333333333334" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "instance": "10.142.0.107:9100" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "instance": "10.142.0.43:9100" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "60756284.2" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "1817.5333333333333" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382434" 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "ds-idrepo-0" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "pod": "ds-cts-2" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "0" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "instance": "10.142.0.107:9100" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "instance": "10.142.0.43:9100" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "0.06046666666666643" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "ds-cts-0" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "pod": "ds-cts-2" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "0" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "15688.119827645167" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "0" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "6.610763005095425" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382434" 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382434" 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382434" 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:33:54 DEBUG "instance": "10.142.0.107:9100" 20:33:54 DEBUG }, 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG [...] 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG "instance": "10.142.0.43:9100" 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:33:54 DEBUG }, 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:54 DEBUG }, 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG "value": [ 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG "value": [ 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG { 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "1826.7333333333333" 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "60750025.63333333" 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG ] 20:33:54 DEBUG "data": { 20:33:54 DEBUG ] 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG } 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG } 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG ] 20:33:54 DEBUG "result": [ 20:33:54 DEBUG ] 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG } 20:33:54 DEBUG { 20:33:54 DEBUG } 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG } 20:33:54 DEBUG "metric": { 20:33:54 DEBUG } 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:33:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:33:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:33:54 DEBUG }, 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG }, 20:33:54 DEBUG }, 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG "value": [ 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG "value": [ 20:33:54 DEBUG "value": [ 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:33:54 DEBUG [...] 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG 20:33:54 DEBUG { 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG { 20:33:54 DEBUG 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG 20:33:54 DEBUG "data": { 20:33:54 DEBUG 20:33:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:33:54 DEBUG "data": { 20:33:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:33:54 DEBUG }, 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG }, 20:33:54 DEBUG "result": [ 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG "result": [ 20:33:54 DEBUG "value": [ 20:33:54 DEBUG { 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG { 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "metric": { 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "0" 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "328723" 20:33:54 DEBUG "instance": "10.142.0.107:9100" 20:33:54 DEBUG "6.65802061392199" 20:33:54 DEBUG ] 20:33:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:33:54 DEBUG ] 20:33:54 DEBUG }, 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG }, 20:33:54 DEBUG } 20:33:54 DEBUG "value": [ 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG "value": [ 20:33:54 DEBUG ] 20:33:54 DEBUG 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG 20:33:54 DEBUG } 20:33:54 DEBUG 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 DEBUG 20:33:54 DEBUG } 20:33:54 DEBUG 20:33:54 DEBUG } 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "instance": "10.142.0.43:9100" 20:33:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:33:54 DEBUG }, 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "96535938048" 20:33:54 DEBUG "0" 20:33:54 DEBUG ] 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "ds-cts-0" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "pod": "ds-cts-2" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "0" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382434" 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "ds-cts-0" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "pod": "ds-cts-2" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "0" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "instance": "10.142.0.107:9100" 20:33:54 DEBUG "pod": "ds-idrepo-0" 20:33:54 DEBUG }, 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "instance": "10.142.0.43:9100" 20:33:54 DEBUG "pod": "ds-idrepo-2" 20:33:54 DEBUG }, 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "0.03566666666666678" 20:33:54 DEBUG "0" 20:33:54 DEBUG ] 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO [http_cmd]: http status code OK 20:33:54 DEBUG --- status code --- 20:33:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:54 DEBUG --- http response --- 20:33:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:54 DEBUG ----- output ----- 20:33:54 DEBUG { 20:33:54 DEBUG "status": "success", 20:33:54 DEBUG "data": { 20:33:54 DEBUG "resultType": "vector", 20:33:54 DEBUG "result": [ 20:33:54 DEBUG { 20:33:54 DEBUG "metric": { 20:33:54 DEBUG "pod": "ds-idrepo-0" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG [...] 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG 20:33:54 DEBUG "pod": "ds-idrepo-2" 20:33:54 DEBUG }, 20:33:54 DEBUG "value": [ 20:33:54 DEBUG 1713382434, 20:33:54 DEBUG "0" 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG ] 20:33:54 DEBUG } 20:33:54 DEBUG } 20:33:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382434" 20:33:54 INFO 20:33:54 INFO [loop_until]: kubectl --namespace=xlou top pods 20:33:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:33:55 INFO [http_cmd]: http status code OK 20:33:55 DEBUG --- status code --- 20:33:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:55 DEBUG --- http response --- 20:33:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:55 DEBUG ----- output ----- 20:33:55 DEBUG { 20:33:55 DEBUG "status": "success", 20:33:55 DEBUG "data": { 20:33:55 DEBUG "resultType": "vector", 20:33:55 DEBUG "result": [ 20:33:55 DEBUG { 20:33:55 DEBUG "metric": { 20:33:55 DEBUG "pod": "ds-idrepo-0" 20:33:55 DEBUG }, 20:33:55 DEBUG "value": [ 20:33:55 DEBUG 20:33:55 DEBUG 20:33:55 DEBUG 20:33:55 DEBUG [...] 20:33:55 DEBUG 20:33:55 DEBUG 20:33:55 DEBUG 20:33:55 DEBUG "pod": "ds-idrepo-2" 20:33:55 DEBUG }, 20:33:55 DEBUG "value": [ 20:33:55 DEBUG 1713382434, 20:33:55 DEBUG "0" 20:33:55 DEBUG ] 20:33:55 DEBUG } 20:33:55 DEBUG ] 20:33:55 DEBUG } 20:33:55 DEBUG } 20:33:55 INFO [loop_until]: OK (rc = 0) 20:33:55 DEBUG --- stdout --- 20:33:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11715m 8889Mi am-b59c765b7-bgsxp 10887m 8715Mi am-b59c765b7-ch5q4 12097m 8851Mi ds-cts-0 12954m 7638Mi ds-cts-1 11370m 7515Mi ds-cts-2 12869m 7812Mi ds-idrepo-0 5447m 3092Mi ds-idrepo-1 5222m 2700Mi ds-idrepo-2 5317m 2607Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 965Mi idm-b85f9d978-wnhxb 5m 1003Mi lodemon-db4967764-9zt9t 7m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6800m 836Mi 20:33:55 DEBUG --- stderr --- 20:33:55 DEBUG 20:33:55 INFO 20:33:55 INFO [loop_until]: kubectl --namespace=xlou top node 20:33:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:33:55 INFO [loop_until]: OK (rc = 0) 20:33:55 DEBUG --- stdout --- 20:33:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 78m 0% 5488Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5187m 17% 5975Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11750m 39% 11425Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 2983Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12555m 42% 9807Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 164m 0% 5010Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11576m 38% 11492Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27946m 93% 2907Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11770m 39% 10553Mi 9% gke-xlou-cdm-ds-0f601751-j488 5558m 18% 5143Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11532m 38% 10910Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5689m 19% 5758Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12957m 43% 9423Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6797m 42% 4627Mi 7% 20:33:55 DEBUG --- stderr --- 20:33:55 DEBUG 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382494" 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382494" 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "976273408" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 ERROR [http_cmd]: ERROR 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 ERROR --- status code --- 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 ERROR --- http response --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:34:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:34:54 DEBUG }, 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:34:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:34:54 DEBUG }, 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "0.0035" 20:34:54 DEBUG "1805" 20:34:54 DEBUG ] 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "1806.1333333333334" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382494" 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "instance": "10.142.0.107:9100" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "instance": "10.142.0.43:9100" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "60699520.03333333" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382494" 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "pod": "ds-idrepo-0" 20:34:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:34:54 DEBUG }, 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "pod": "ds-cts-2" 20:34:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:34:54 DEBUG }, 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "0" 20:34:54 DEBUG "1824.2" 20:34:54 DEBUG ] 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "instance": "10.142.0.107:9100" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "instance": "10.142.0.43:9100" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "0.1508333333333335" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "pod": "ds-cts-0" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "pod": "ds-cts-2" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "0" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "0" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "23945.168660126325" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382494" 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "6.645523469560548" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "6.681047265091964" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382494" 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "instance": "10.142.0.107:9100" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "instance": "10.142.0.43:9100" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "99448159232" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382494" 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:34:54 DEBUG }, 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:54 DEBUG "value": [ 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG { 20:34:54 DEBUG "1805.1" 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG ] 20:34:54 DEBUG "data": { 20:34:54 DEBUG } 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG ] 20:34:54 DEBUG "result": [ 20:34:54 DEBUG } 20:34:54 DEBUG { 20:34:54 DEBUG } 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "instance": "10.142.0.107:9100" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "instance": "10.142.0.43:9100" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "60693004.1" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "329125" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "0" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "0" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 INFO [http_cmd]: http status code OK 20:34:54 DEBUG --- status code --- 20:34:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:54 DEBUG --- http response --- 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382494" 20:34:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:34:54 DEBUG ----- output ----- 20:34:54 DEBUG { 20:34:54 DEBUG "status": "success", 20:34:54 DEBUG "data": { 20:34:54 DEBUG "resultType": "vector", 20:34:54 DEBUG "result": [ 20:34:54 DEBUG { 20:34:54 DEBUG "metric": { 20:34:54 DEBUG "pod": "ds-cts-0" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG [...] 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG 20:34:54 DEBUG "pod": "ds-cts-2" 20:34:54 DEBUG }, 20:34:54 DEBUG "value": [ 20:34:54 DEBUG 1713382494, 20:34:54 DEBUG "25.7" 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG ] 20:34:54 DEBUG } 20:34:54 DEBUG } 20:34:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382494" 20:34:55 INFO [http_cmd]: http status code OK 20:34:55 DEBUG --- status code --- 20:34:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:55 DEBUG --- http response --- 20:34:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:34:55 DEBUG ----- output ----- 20:34:55 DEBUG { 20:34:55 DEBUG "status": "success", 20:34:55 DEBUG "data": { 20:34:55 DEBUG "resultType": "vector", 20:34:55 DEBUG "result": [ 20:34:55 DEBUG { 20:34:55 DEBUG "metric": { 20:34:55 DEBUG "pod": "ds-cts-0" 20:34:55 DEBUG }, 20:34:55 DEBUG "value": [ 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG [...] 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG "pod": "ds-cts-2" 20:34:55 DEBUG }, 20:34:55 DEBUG "value": [ 20:34:55 DEBUG 1713382494, 20:34:55 DEBUG "0" 20:34:55 DEBUG ] 20:34:55 DEBUG } 20:34:55 DEBUG ] 20:34:55 DEBUG } 20:34:55 DEBUG } 20:34:55 INFO [http_cmd]: http status code OK 20:34:55 DEBUG --- status code --- 20:34:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:55 DEBUG --- http response --- 20:34:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:55 DEBUG ----- output ----- 20:34:55 DEBUG { 20:34:55 DEBUG "status": "success", 20:34:55 DEBUG "data": { 20:34:55 DEBUG "resultType": "vector", 20:34:55 DEBUG "result": [ 20:34:55 DEBUG { 20:34:55 DEBUG "metric": { 20:34:55 DEBUG "instance": "10.142.0.107:9100" 20:34:55 DEBUG }, 20:34:55 DEBUG "value": [ 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG [...] 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG "instance": "10.142.0.43:9100" 20:34:55 DEBUG }, 20:34:55 DEBUG "value": [ 20:34:55 DEBUG 1713382494, 20:34:55 DEBUG "0.07933333333333341" 20:34:55 DEBUG ] 20:34:55 DEBUG } 20:34:55 DEBUG ] 20:34:55 DEBUG } 20:34:55 DEBUG } 20:34:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382495" 20:34:55 INFO [http_cmd]: http status code OK 20:34:55 DEBUG --- status code --- 20:34:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:55 DEBUG --- http response --- 20:34:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:55 DEBUG ----- output ----- 20:34:55 DEBUG { 20:34:55 DEBUG "status": "success", 20:34:55 DEBUG "data": { 20:34:55 DEBUG "resultType": "vector", 20:34:55 DEBUG "result": [ 20:34:55 DEBUG { 20:34:55 DEBUG "metric": { 20:34:55 DEBUG "pod": "ds-idrepo-0" 20:34:55 DEBUG }, 20:34:55 DEBUG "value": [ 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG [...] 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG "pod": "ds-idrepo-2" 20:34:55 DEBUG }, 20:34:55 DEBUG "value": [ 20:34:55 DEBUG 1713382495, 20:34:55 DEBUG "0" 20:34:55 DEBUG ] 20:34:55 DEBUG } 20:34:55 DEBUG ] 20:34:55 DEBUG } 20:34:55 DEBUG } 20:34:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382495" 20:34:55 INFO [http_cmd]: http status code OK 20:34:55 DEBUG --- status code --- 20:34:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:55 DEBUG --- http response --- 20:34:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:55 DEBUG ----- output ----- 20:34:55 DEBUG { 20:34:55 DEBUG "status": "success", 20:34:55 DEBUG "data": { 20:34:55 DEBUG "resultType": "vector", 20:34:55 DEBUG "result": [ 20:34:55 DEBUG { 20:34:55 DEBUG "metric": { 20:34:55 DEBUG "pod": "ds-idrepo-0" 20:34:55 DEBUG }, 20:34:55 DEBUG "value": [ 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG [...] 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG "pod": "ds-idrepo-2" 20:34:55 DEBUG }, 20:34:55 DEBUG "value": [ 20:34:55 DEBUG 1713382495, 20:34:55 DEBUG "0" 20:34:55 DEBUG ] 20:34:55 DEBUG } 20:34:55 DEBUG ] 20:34:55 DEBUG } 20:34:55 DEBUG } 20:34:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382495" 20:34:55 INFO 20:34:55 INFO [loop_until]: kubectl --namespace=xlou top pods 20:34:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:34:55 INFO [http_cmd]: http status code OK 20:34:55 DEBUG --- status code --- 20:34:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:55 DEBUG --- http response --- 20:34:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:55 DEBUG ----- output ----- 20:34:55 DEBUG { 20:34:55 DEBUG "status": "success", 20:34:55 DEBUG "data": { 20:34:55 DEBUG "resultType": "vector", 20:34:55 DEBUG "result": [ 20:34:55 DEBUG { 20:34:55 DEBUG "metric": { 20:34:55 DEBUG "pod": "ds-idrepo-0" 20:34:55 DEBUG }, 20:34:55 DEBUG "value": [ 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG [...] 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG 20:34:55 DEBUG "pod": "ds-idrepo-2" 20:34:55 DEBUG }, 20:34:55 DEBUG "value": [ 20:34:55 DEBUG 1713382495, 20:34:55 DEBUG "0" 20:34:55 DEBUG ] 20:34:55 DEBUG } 20:34:55 DEBUG ] 20:34:55 DEBUG } 20:34:55 DEBUG } 20:34:55 INFO [loop_until]: OK (rc = 0) 20:34:55 DEBUG --- stdout --- 20:34:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11969m 8891Mi am-b59c765b7-bgsxp 10939m 8715Mi am-b59c765b7-ch5q4 11881m 8852Mi ds-cts-0 12485m 7644Mi ds-cts-1 11480m 7520Mi ds-cts-2 13275m 7811Mi ds-idrepo-0 5214m 3093Mi ds-idrepo-1 5210m 2716Mi ds-idrepo-2 5275m 2608Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 9m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6635m 838Mi 20:34:55 DEBUG --- stderr --- 20:34:55 DEBUG 20:34:55 INFO 20:34:55 INFO [loop_until]: kubectl --namespace=xlou top node 20:34:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:34:55 INFO [loop_until]: OK (rc = 0) 20:34:55 DEBUG --- stdout --- 20:34:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 5495Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5119m 17% 5968Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11853m 39% 11425Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 2983Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12654m 42% 9821Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5021Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11977m 40% 11497Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27310m 91% 2912Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11828m 39% 10547Mi 9% gke-xlou-cdm-ds-0f601751-j488 5569m 18% 5142Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11115m 37% 10925Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5742m 19% 5765Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12540m 42% 9427Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6828m 42% 4627Mi 7% 20:34:55 DEBUG --- stderr --- 20:34:55 DEBUG 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382554" 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382554" 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382554" 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "0.0055" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "1818.2" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382554" 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382554" 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382554" 20:35:54 ERROR [http_cmd]: ERROR 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 ERROR --- status code --- 20:35:54 DEBUG --- status code --- 20:35:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 ERROR --- http response --- 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:35:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "1818.0333333333333" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "978288640" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382554" 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382554" 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382554" 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "instance": "10.142.0.107:9100" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "instance": "10.142.0.43:9100" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "60739974.766666666" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "instance": "10.142.0.107:9100" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "instance": "10.142.0.43:9100" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "0.0608333333333339" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382554" 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "1821.1666666666667" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "pod": "ds-idrepo-0" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "pod": "ds-cts-2" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "0" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382554" 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382554" 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "0" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "pod": "ds-cts-0" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "pod": "ds-cts-2" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "0.000017104444444444444" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382554" 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "pod": "ds-cts-0" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "0" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382554" 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382554" 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382554" 20:35:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382554" 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "instance": "10.142.0.107:9100" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "instance": "10.142.0.43:9100" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "102225566720" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "pod": "ds-cts-0" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "0" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "1817.5" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:54 INFO [http_cmd]: http status code OK 20:35:54 DEBUG --- status code --- 20:35:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:54 DEBUG --- http response --- 20:35:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:54 DEBUG ----- output ----- 20:35:54 DEBUG { 20:35:54 DEBUG "status": "success", 20:35:54 DEBUG "data": { 20:35:54 DEBUG "resultType": "vector", 20:35:54 DEBUG "result": [ 20:35:54 DEBUG { 20:35:54 DEBUG "metric": { 20:35:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG [...] 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG 20:35:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:35:54 DEBUG }, 20:35:54 DEBUG "value": [ 20:35:54 DEBUG 1713382554, 20:35:54 DEBUG "6.670924174900477" 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG ] 20:35:54 DEBUG } 20:35:54 DEBUG } 20:35:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382555" 20:35:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382555" 20:35:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382555" 20:35:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382555" 20:35:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382555" 20:35:55 INFO [http_cmd]: http status code OK 20:35:55 DEBUG --- status code --- 20:35:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:55 DEBUG --- http response --- 20:35:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:55 DEBUG ----- output ----- 20:35:55 DEBUG { 20:35:55 DEBUG "status": "success", 20:35:55 DEBUG "data": { 20:35:55 DEBUG "resultType": "vector", 20:35:55 DEBUG "result": [ 20:35:55 DEBUG { 20:35:55 DEBUG "metric": { 20:35:55 DEBUG "instance": "10.142.0.107:9100" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG [...] 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG "instance": "10.142.0.43:9100" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 1713382555, 20:35:55 DEBUG "60732624.06666666" 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG } 20:35:55 INFO [http_cmd]: http status code OK 20:35:55 DEBUG --- status code --- 20:35:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:55 DEBUG --- http response --- 20:35:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:55 DEBUG ----- output ----- 20:35:55 DEBUG { 20:35:55 DEBUG "status": "success", 20:35:55 DEBUG "data": { 20:35:55 DEBUG "resultType": "vector", 20:35:55 DEBUG "result": [ 20:35:55 DEBUG { 20:35:55 DEBUG "metric": { 20:35:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG [...] 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 1713382555, 20:35:55 DEBUG "328566.5" 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG } 20:35:55 INFO [http_cmd]: http status code OK 20:35:55 INFO [http_cmd]: http status code OK 20:35:55 INFO [http_cmd]: http status code OK 20:35:55 DEBUG --- status code --- 20:35:55 DEBUG --- status code --- 20:35:55 DEBUG --- status code --- 20:35:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:55 DEBUG --- http response --- 20:35:55 DEBUG --- http response --- 20:35:55 DEBUG --- http response --- 20:35:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:35:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:55 DEBUG ----- output ----- 20:35:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:35:55 DEBUG ----- output ----- 20:35:55 DEBUG { 20:35:55 DEBUG ----- output ----- 20:35:55 DEBUG { 20:35:55 DEBUG "status": "success", 20:35:55 DEBUG { 20:35:55 DEBUG "status": "success", 20:35:55 DEBUG "data": { 20:35:55 DEBUG "status": "success", 20:35:55 DEBUG "data": { 20:35:55 DEBUG "resultType": "vector", 20:35:55 DEBUG "data": { 20:35:55 DEBUG "resultType": "vector", 20:35:55 DEBUG "result": [ 20:35:55 DEBUG "resultType": "vector", 20:35:55 DEBUG "result": [ 20:35:55 DEBUG { 20:35:55 DEBUG "result": [ 20:35:55 DEBUG { 20:35:55 DEBUG "metric": { 20:35:55 DEBUG { 20:35:55 DEBUG "metric": { 20:35:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:35:55 DEBUG "metric": { 20:35:55 DEBUG "instance": "10.142.0.107:9100" 20:35:55 DEBUG }, 20:35:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG [...] 20:35:55 DEBUG 20:35:55 DEBUG [...] 20:35:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382555" 20:35:55 DEBUG 20:35:55 DEBUG [...] 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:35:55 DEBUG 20:35:55 DEBUG "instance": "10.142.0.43:9100" 20:35:55 DEBUG }, 20:35:55 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 1713382555, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 1713382555, 20:35:55 DEBUG "0" 20:35:55 DEBUG 1713382555, 20:35:55 DEBUG "0.04499999999999994" 20:35:55 DEBUG ] 20:35:55 DEBUG "0" 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG } 20:35:55 DEBUG } 20:35:55 DEBUG } 20:35:55 DEBUG } 20:35:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382555" 20:35:55 INFO [http_cmd]: http status code OK 20:35:55 DEBUG --- status code --- 20:35:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:55 DEBUG --- http response --- 20:35:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:35:55 DEBUG ----- output ----- 20:35:55 DEBUG { 20:35:55 DEBUG "status": "success", 20:35:55 DEBUG "data": { 20:35:55 DEBUG "resultType": "vector", 20:35:55 DEBUG "result": [ 20:35:55 DEBUG { 20:35:55 DEBUG "metric": { 20:35:55 DEBUG "pod": "ds-cts-0" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG [...] 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG "pod": "ds-cts-2" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 1713382555, 20:35:55 DEBUG "30" 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG } 20:35:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382555" 20:35:55 INFO [http_cmd]: http status code OK 20:35:55 DEBUG --- status code --- 20:35:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:55 DEBUG --- http response --- 20:35:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:35:55 DEBUG ----- output ----- 20:35:55 DEBUG { 20:35:55 DEBUG "status": "success", 20:35:55 DEBUG "data": { 20:35:55 DEBUG "resultType": "vector", 20:35:55 DEBUG "result": [ 20:35:55 DEBUG { 20:35:55 DEBUG "metric": { 20:35:55 DEBUG "pod": "ds-cts-0" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG [...] 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG "pod": "ds-cts-2" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 1713382555, 20:35:55 DEBUG "0" 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG } 20:35:55 INFO [http_cmd]: http status code OK 20:35:55 DEBUG --- status code --- 20:35:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:55 DEBUG --- http response --- 20:35:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:55 DEBUG ----- output ----- 20:35:55 DEBUG { 20:35:55 DEBUG "status": "success", 20:35:55 DEBUG "data": { 20:35:55 DEBUG "resultType": "vector", 20:35:55 DEBUG "result": [ 20:35:55 DEBUG { 20:35:55 DEBUG "metric": { 20:35:55 DEBUG "pod": "ds-idrepo-0" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG [...] 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG "pod": "ds-idrepo-2" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 1713382555, 20:35:55 DEBUG "0" 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG } 20:35:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382555" 20:35:55 INFO [http_cmd]: http status code OK 20:35:55 DEBUG --- status code --- 20:35:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:55 DEBUG --- http response --- 20:35:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:55 DEBUG ----- output ----- 20:35:55 DEBUG { 20:35:55 DEBUG "status": "success", 20:35:55 DEBUG "data": { 20:35:55 DEBUG "resultType": "vector", 20:35:55 DEBUG "result": [ 20:35:55 DEBUG { 20:35:55 DEBUG "metric": { 20:35:55 DEBUG "pod": "ds-idrepo-0" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG [...] 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG "pod": "ds-idrepo-2" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 1713382555, 20:35:55 DEBUG "0" 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG } 20:35:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382555" 20:35:55 INFO [http_cmd]: http status code OK 20:35:55 DEBUG --- status code --- 20:35:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:55 DEBUG --- http response --- 20:35:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:55 DEBUG ----- output ----- 20:35:55 DEBUG { 20:35:55 DEBUG "status": "success", 20:35:55 DEBUG "data": { 20:35:55 DEBUG "resultType": "vector", 20:35:55 DEBUG "result": [ 20:35:55 DEBUG { 20:35:55 DEBUG "metric": { 20:35:55 DEBUG "pod": "ds-idrepo-0" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG [...] 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG 20:35:55 DEBUG "pod": "ds-idrepo-2" 20:35:55 DEBUG }, 20:35:55 DEBUG "value": [ 20:35:55 DEBUG 1713382555, 20:35:55 DEBUG "0" 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG ] 20:35:55 DEBUG } 20:35:55 DEBUG } 20:35:55 INFO 20:35:55 INFO [loop_until]: kubectl --namespace=xlou top pods 20:35:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:35:55 INFO [loop_until]: OK (rc = 0) 20:35:55 DEBUG --- stdout --- 20:35:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11900m 8892Mi am-b59c765b7-bgsxp 10437m 8715Mi am-b59c765b7-ch5q4 11574m 8850Mi ds-cts-0 12707m 7658Mi ds-cts-1 11381m 7536Mi ds-cts-2 13465m 7794Mi ds-idrepo-0 5199m 3093Mi ds-idrepo-1 5228m 2716Mi ds-idrepo-2 5241m 2689Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 9m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6621m 840Mi 20:35:55 DEBUG --- stderr --- 20:35:55 DEBUG 20:35:55 INFO 20:35:55 INFO [loop_until]: kubectl --namespace=xlou top node 20:35:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:35:55 INFO [loop_until]: OK (rc = 0) 20:35:55 DEBUG --- stdout --- 20:35:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 5484Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5130m 17% 5973Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11446m 38% 11422Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 2984Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12679m 42% 9825Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 170m 0% 5012Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11983m 40% 11493Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27994m 93% 2923Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11694m 39% 10553Mi 9% gke-xlou-cdm-ds-0f601751-j488 5465m 18% 5206Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11272m 37% 10940Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5653m 18% 5775Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13018m 43% 9421Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6589m 41% 4626Mi 7% 20:35:55 DEBUG --- stderr --- 20:35:55 DEBUG 20:36:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382614" 20:36:54 INFO [http_cmd]: http status code OK 20:36:54 DEBUG --- status code --- 20:36:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:54 DEBUG --- http response --- 20:36:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:36:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382614" 20:36:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382614" 20:36:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382614" 20:36:54 INFO [http_cmd]: http status code OK 20:36:54 DEBUG --- status code --- 20:36:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:54 DEBUG --- http response --- 20:36:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:54 DEBUG ----- output ----- 20:36:54 DEBUG { 20:36:54 DEBUG "status": "success", 20:36:54 DEBUG "data": { 20:36:54 DEBUG "resultType": "vector", 20:36:54 DEBUG "result": [ 20:36:54 DEBUG { 20:36:54 DEBUG "metric": { 20:36:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG [...] 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 1713382614, 20:36:54 DEBUG "1830.9666666666667" 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG } 20:36:54 INFO [http_cmd]: http status code OK 20:36:54 DEBUG --- status code --- 20:36:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:54 DEBUG --- http response --- 20:36:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:54 DEBUG ----- output ----- 20:36:54 DEBUG { 20:36:54 DEBUG "status": "success", 20:36:54 DEBUG "data": { 20:36:54 DEBUG "resultType": "vector", 20:36:54 DEBUG "result": [ 20:36:54 DEBUG { 20:36:54 DEBUG "metric": { 20:36:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG [...] 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 1713382614, 20:36:54 DEBUG "0.006" 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG } 20:36:54 INFO [http_cmd]: http status code OK 20:36:54 DEBUG --- status code --- 20:36:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:54 DEBUG --- http response --- 20:36:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:54 DEBUG ----- output ----- 20:36:54 DEBUG { 20:36:54 DEBUG "status": "success", 20:36:54 DEBUG "data": { 20:36:54 DEBUG "resultType": "vector", 20:36:54 DEBUG "result": [ 20:36:54 DEBUG { 20:36:54 DEBUG "metric": { 20:36:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG [...] 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 1713382614, 20:36:54 DEBUG "980668416" 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG } 20:36:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382614" 20:36:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382614" 20:36:54 INFO [http_cmd]: http status code OK 20:36:54 DEBUG --- status code --- 20:36:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:54 DEBUG --- http response --- 20:36:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:54 DEBUG ----- output ----- 20:36:54 DEBUG { 20:36:54 DEBUG "status": "success", 20:36:54 DEBUG "data": { 20:36:54 DEBUG "resultType": "vector", 20:36:54 DEBUG "result": [ 20:36:54 DEBUG { 20:36:54 DEBUG "metric": { 20:36:54 DEBUG "instance": "10.142.0.107:9100" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG [...] 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG "instance": "10.142.0.43:9100" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 1713382614, 20:36:54 DEBUG "60742948.766666666" 20:36:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382614" 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG } 20:36:54 ERROR [http_cmd]: ERROR 20:36:54 ERROR --- status code --- 20:36:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:36:54 ERROR --- http response --- 20:36:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:36:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:36:54 INFO [http_cmd]: http status code OK 20:36:54 DEBUG --- status code --- 20:36:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:54 DEBUG --- http response --- 20:36:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:54 DEBUG ----- output ----- 20:36:54 DEBUG { 20:36:54 DEBUG "status": "success", 20:36:54 DEBUG "data": { 20:36:54 DEBUG "resultType": "vector", 20:36:54 DEBUG "result": [ 20:36:54 DEBUG { 20:36:54 DEBUG "metric": { 20:36:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG [...] 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 1713382614, 20:36:54 DEBUG "1831.2" 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG } 20:36:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382614" 20:36:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382614" 20:36:54 INFO [http_cmd]: http status code OK 20:36:54 DEBUG --- status code --- 20:36:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:54 DEBUG --- http response --- 20:36:54 INFO [http_cmd]: http status code OK 20:36:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:54 DEBUG --- status code --- 20:36:54 DEBUG ----- output ----- 20:36:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:54 DEBUG { 20:36:54 DEBUG --- http response --- 20:36:54 DEBUG "status": "success", 20:36:54 DEBUG "data": { 20:36:54 DEBUG "resultType": "vector", 20:36:54 DEBUG "result": [ 20:36:54 DEBUG { 20:36:54 DEBUG "metric": { 20:36:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG [...] 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 1713382614, 20:36:54 DEBUG "1827.2666666666667" 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG } 20:36:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:54 DEBUG ----- output ----- 20:36:54 DEBUG { 20:36:54 DEBUG "status": "success", 20:36:54 DEBUG "data": { 20:36:54 DEBUG "resultType": "vector", 20:36:54 DEBUG "result": [ 20:36:54 DEBUG { 20:36:54 DEBUG "metric": { 20:36:54 DEBUG "instance": "10.142.0.107:9100" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG [...] 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG "instance": "10.142.0.43:9100" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 1713382614, 20:36:54 DEBUG "0.15156666666666657" 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG } 20:36:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382614" 20:36:54 INFO [http_cmd]: http status code OK 20:36:54 DEBUG --- status code --- 20:36:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:54 DEBUG --- http response --- 20:36:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:36:54 DEBUG ----- output ----- 20:36:54 DEBUG { 20:36:54 DEBUG "status": "success", 20:36:54 DEBUG "data": { 20:36:54 DEBUG "resultType": "vector", 20:36:54 DEBUG "result": [ 20:36:54 DEBUG { 20:36:54 DEBUG "metric": { 20:36:54 DEBUG "pod": "ds-idrepo-0" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG [...] 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG "pod": "ds-cts-2" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 1713382614, 20:36:54 DEBUG "0" 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG } 20:36:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382614" 20:36:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382614" 20:36:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382614" 20:36:54 INFO [http_cmd]: http status code OK 20:36:54 INFO [http_cmd]: http status code OK 20:36:54 DEBUG --- status code --- 20:36:54 DEBUG --- status code --- 20:36:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:54 DEBUG --- http response --- 20:36:54 DEBUG --- http response --- 20:36:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:54 DEBUG ----- output ----- 20:36:54 DEBUG { 20:36:54 DEBUG "status": "success", 20:36:54 DEBUG "data": { 20:36:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:36:54 DEBUG "resultType": "vector", 20:36:54 DEBUG ----- output ----- 20:36:54 DEBUG "result": [ 20:36:54 DEBUG { 20:36:54 DEBUG { 20:36:54 DEBUG "status": "success", 20:36:54 DEBUG "metric": { 20:36:54 DEBUG "data": { 20:36:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:36:54 DEBUG "resultType": "vector", 20:36:54 DEBUG }, 20:36:54 DEBUG "result": [ 20:36:54 DEBUG "value": [ 20:36:54 DEBUG { 20:36:54 DEBUG 20:36:54 DEBUG "metric": { 20:36:54 DEBUG 20:36:54 DEBUG "pod": "ds-cts-0" 20:36:54 DEBUG 20:36:54 DEBUG }, 20:36:54 DEBUG [...] 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:36:54 DEBUG [...] 20:36:54 DEBUG }, 20:36:54 DEBUG 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 20:36:54 DEBUG 1713382614, 20:36:54 DEBUG 20:36:54 DEBUG "0" 20:36:54 DEBUG "pod": "ds-cts-2" 20:36:54 DEBUG ] 20:36:54 DEBUG }, 20:36:54 DEBUG } 20:36:54 DEBUG "value": [ 20:36:54 DEBUG ] 20:36:54 DEBUG 1713382614, 20:36:54 DEBUG } 20:36:54 DEBUG "0.000033333333333333335" 20:36:54 DEBUG } 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG } 20:36:54 INFO [http_cmd]: http status code OK 20:36:54 DEBUG --- status code --- 20:36:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:54 DEBUG --- http response --- 20:36:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:36:54 DEBUG ----- output ----- 20:36:54 DEBUG { 20:36:54 DEBUG "status": "success", 20:36:54 DEBUG "data": { 20:36:54 DEBUG "resultType": "vector", 20:36:54 DEBUG "result": [ 20:36:54 DEBUG { 20:36:54 DEBUG "metric": { 20:36:54 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG [...] 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG 20:36:54 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:36:54 DEBUG }, 20:36:54 DEBUG "value": [ 20:36:54 DEBUG 1713382614, 20:36:54 DEBUG "57438.572065301894" 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG ] 20:36:54 DEBUG } 20:36:54 DEBUG } 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382615" 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382615" 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382615" 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382615" 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG { 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "1831.0333333333333" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG { 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "4.43872191132329" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG { 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "instance": "10.142.0.107:9100" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "instance": "10.142.0.43:9100" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "104913998848" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG { 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "6.652803076018792" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382615" 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382615" 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382615" 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382615" 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG { 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "0" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG { 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "pod": "ds-idrepo-0" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "pod": "ds-idrepo-2" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "0" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG { 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "instance": "10.142.0.107:9100" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "instance": "10.142.0.43:9100" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "60735826.766666666" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382615" 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382615" 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG { 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "327986.5" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382615" 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG { 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "0" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG { 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "instance": "10.142.0.107:9100" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "instance": "10.142.0.43:9100" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "0.07266666666666664" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382615" 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382615" 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG { 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "pod": "ds-cts-0" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "pod": "ds-cts-2" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "30" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG { 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "pod": "ds-idrepo-0" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "pod": "ds-idrepo-2" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "0" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG { 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "pod": "ds-cts-0" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "pod": "ds-cts-2" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "0" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382615" 20:36:55 INFO [http_cmd]: http status code OK 20:36:55 DEBUG --- status code --- 20:36:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:55 DEBUG --- http response --- 20:36:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:55 DEBUG ----- output ----- 20:36:55 DEBUG { 20:36:55 DEBUG "status": "success", 20:36:55 DEBUG "data": { 20:36:55 DEBUG "resultType": "vector", 20:36:55 DEBUG "result": [ 20:36:55 DEBUG { 20:36:55 DEBUG "metric": { 20:36:55 DEBUG "pod": "ds-idrepo-0" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG [...] 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG 20:36:55 DEBUG "pod": "ds-idrepo-2" 20:36:55 DEBUG }, 20:36:55 DEBUG "value": [ 20:36:55 DEBUG 1713382615, 20:36:55 DEBUG "0" 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG ] 20:36:55 DEBUG } 20:36:55 DEBUG } 20:36:55 INFO 20:36:55 INFO [loop_until]: kubectl --namespace=xlou top pods 20:36:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:55 INFO [loop_until]: OK (rc = 0) 20:36:55 DEBUG --- stdout --- 20:36:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11902m 8893Mi am-b59c765b7-bgsxp 10548m 8709Mi am-b59c765b7-ch5q4 11552m 8852Mi ds-cts-0 12587m 7662Mi ds-cts-1 11624m 7540Mi ds-cts-2 13174m 7825Mi ds-idrepo-0 5195m 3092Mi ds-idrepo-1 5073m 2717Mi ds-idrepo-2 5097m 2690Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6606m 841Mi 20:36:55 DEBUG --- stderr --- 20:36:55 DEBUG 20:36:55 INFO 20:36:55 INFO [loop_until]: kubectl --namespace=xlou top node 20:36:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:55 INFO [loop_until]: OK (rc = 0) 20:36:55 DEBUG --- stdout --- 20:36:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 80m 0% 5497Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5064m 16% 5981Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11780m 39% 11417Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 2981Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12434m 41% 9832Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 5015Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11879m 39% 11495Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27747m 92% 2912Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11627m 38% 10542Mi 9% gke-xlou-cdm-ds-0f601751-j488 5449m 18% 5218Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11313m 37% 10931Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5625m 18% 5777Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12918m 43% 9431Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6794m 42% 4630Mi 7% 20:36:55 DEBUG --- stderr --- 20:36:55 DEBUG 20:37:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382674" 20:37:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382674" 20:37:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382674" 20:37:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382674" 20:37:54 INFO [http_cmd]: http status code OK 20:37:54 DEBUG --- status code --- 20:37:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:54 DEBUG --- http response --- 20:37:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:37:54 INFO [http_cmd]: http status code OK 20:37:54 DEBUG --- status code --- 20:37:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:54 DEBUG --- http response --- 20:37:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:54 DEBUG ----- output ----- 20:37:54 DEBUG { 20:37:54 DEBUG "status": "success", 20:37:54 DEBUG "data": { 20:37:54 DEBUG "resultType": "vector", 20:37:54 DEBUG "result": [ 20:37:54 DEBUG { 20:37:54 DEBUG "metric": { 20:37:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:37:54 DEBUG }, 20:37:54 DEBUG "value": [ 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG [...] 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:37:54 DEBUG }, 20:37:54 DEBUG "value": [ 20:37:54 DEBUG 1713382674, 20:37:54 DEBUG "984793088" 20:37:54 DEBUG ] 20:37:54 DEBUG } 20:37:54 DEBUG ] 20:37:54 DEBUG } 20:37:54 DEBUG } 20:37:54 INFO [http_cmd]: http status code OK 20:37:54 INFO [http_cmd]: http status code OK 20:37:54 DEBUG --- status code --- 20:37:54 DEBUG --- status code --- 20:37:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:54 DEBUG --- http response --- 20:37:54 DEBUG --- http response --- 20:37:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:54 DEBUG ----- output ----- 20:37:54 DEBUG { 20:37:54 DEBUG "status": "success", 20:37:54 DEBUG "data": { 20:37:54 DEBUG "resultType": "vector", 20:37:54 DEBUG "result": [ 20:37:54 DEBUG { 20:37:54 DEBUG "metric": { 20:37:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:37:54 DEBUG }, 20:37:54 DEBUG "value": [ 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG [...] 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:37:54 DEBUG }, 20:37:54 DEBUG "value": [ 20:37:54 DEBUG 1713382674, 20:37:54 DEBUG "0.006500000000000001" 20:37:54 DEBUG ] 20:37:54 DEBUG } 20:37:54 DEBUG ] 20:37:54 DEBUG } 20:37:54 DEBUG } 20:37:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:54 DEBUG ----- output ----- 20:37:54 DEBUG { 20:37:54 DEBUG "status": "success", 20:37:54 DEBUG "data": { 20:37:54 DEBUG "resultType": "vector", 20:37:54 DEBUG "result": [ 20:37:54 DEBUG { 20:37:54 DEBUG "metric": { 20:37:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:37:54 DEBUG }, 20:37:54 DEBUG "value": [ 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG [...] 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:37:54 DEBUG }, 20:37:54 DEBUG "value": [ 20:37:54 DEBUG 1713382674, 20:37:54 DEBUG "1845.4" 20:37:54 DEBUG ] 20:37:54 DEBUG } 20:37:54 DEBUG ] 20:37:54 DEBUG } 20:37:54 DEBUG } 20:37:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382674" 20:37:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382674" 20:37:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382674" 20:37:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382674" 20:37:54 INFO [http_cmd]: http status code OK 20:37:54 DEBUG --- status code --- 20:37:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:54 DEBUG --- http response --- 20:37:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:54 DEBUG ----- output ----- 20:37:54 DEBUG { 20:37:54 DEBUG "status": "success", 20:37:54 DEBUG "data": { 20:37:54 DEBUG "resultType": "vector", 20:37:54 DEBUG "result": [ 20:37:54 DEBUG { 20:37:54 DEBUG "metric": { 20:37:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:37:54 DEBUG }, 20:37:54 DEBUG "value": [ 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG [...] 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:37:54 DEBUG }, 20:37:54 DEBUG "value": [ 20:37:54 DEBUG 1713382674, 20:37:54 DEBUG "1845" 20:37:54 DEBUG ] 20:37:54 DEBUG } 20:37:54 DEBUG ] 20:37:54 DEBUG } 20:37:54 DEBUG } 20:37:54 INFO [http_cmd]: http status code OK 20:37:54 DEBUG --- status code --- 20:37:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:54 DEBUG --- http response --- 20:37:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:54 DEBUG ----- output ----- 20:37:54 DEBUG { 20:37:54 DEBUG "status": "success", 20:37:54 DEBUG "data": { 20:37:54 DEBUG "resultType": "vector", 20:37:54 DEBUG "result": [ 20:37:54 DEBUG { 20:37:54 DEBUG "metric": { 20:37:54 DEBUG "instance": "10.142.0.107:9100" 20:37:54 DEBUG }, 20:37:54 DEBUG "value": [ 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG [...] 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG "instance": "10.142.0.43:9100" 20:37:54 DEBUG }, 20:37:54 DEBUG "value": [ 20:37:54 DEBUG 1713382674, 20:37:54 DEBUG "60938789.83333333" 20:37:54 DEBUG ] 20:37:54 DEBUG } 20:37:54 DEBUG ] 20:37:54 DEBUG } 20:37:54 DEBUG } 20:37:54 INFO [http_cmd]: http status code OK 20:37:54 DEBUG --- status code --- 20:37:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:54 DEBUG --- http response --- 20:37:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:37:54 DEBUG ----- output ----- 20:37:54 DEBUG { 20:37:54 DEBUG "status": "success", 20:37:54 DEBUG "data": { 20:37:54 DEBUG "resultType": "vector", 20:37:54 DEBUG "result": [ 20:37:54 DEBUG { 20:37:54 DEBUG "metric": { 20:37:54 DEBUG "pod": "ds-idrepo-0" 20:37:54 DEBUG }, 20:37:54 DEBUG "value": [ 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG [...] 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG 20:37:54 DEBUG "pod": "ds-cts-2" 20:37:54 DEBUG }, 20:37:54 DEBUG "value": [ 20:37:54 DEBUG 1713382674, 20:37:54 DEBUG "0" 20:37:54 DEBUG ] 20:37:54 DEBUG } 20:37:54 DEBUG ] 20:37:54 DEBUG } 20:37:54 DEBUG } 20:37:54 ERROR [http_cmd]: ERROR 20:37:54 ERROR --- status code --- 20:37:54 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:37:54 ERROR --- http response --- 20:37:54 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:37:54 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:37:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382674" 20:37:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382674" 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "instance": "10.142.0.107:9100" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "instance": "10.142.0.43:9100" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382674, 20:37:55 DEBUG "0.06046666666666643" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382675" 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382674, 20:37:55 DEBUG "0" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "ds-cts-0" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "ds-cts-2" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "0.00015794666666666665" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382675" 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "1804.7666666666667" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382675" 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382675" 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "234314724.01839405" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "6.256802996311593" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382675" 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382675" 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382675" 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:37:55 DEBUG "instance": "10.142.0.107:9100" 20:37:55 DEBUG }, 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:37:55 DEBUG "instance": "10.142.0.43:9100" 20:37:55 DEBUG }, 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "1845.1" 20:37:55 DEBUG "107590040576" 20:37:55 DEBUG ] 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "6.667379416060904" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382675" 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382675" 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "instance": "10.142.0.107:9100" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "instance": "10.142.0.43:9100" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "60930480.36666667" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382675" 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "0" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382675" 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "ds-idrepo-0" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "ds-idrepo-2" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "0" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382675" 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382675" 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "327900" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382675" 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "0" 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382675" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382675" 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "instance": "10.142.0.107:9100" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "instance": "10.142.0.43:9100" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "0.04166666666666662" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "ds-cts-0" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "ds-cts-2" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "30" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "ds-idrepo-0" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "ds-idrepo-2" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "0" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "ds-cts-0" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "ds-cts-2" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "0" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382675" 20:37:55 INFO [http_cmd]: http status code OK 20:37:55 DEBUG --- status code --- 20:37:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:55 DEBUG --- http response --- 20:37:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:55 DEBUG ----- output ----- 20:37:55 DEBUG { 20:37:55 DEBUG "status": "success", 20:37:55 DEBUG "data": { 20:37:55 DEBUG "resultType": "vector", 20:37:55 DEBUG "result": [ 20:37:55 DEBUG { 20:37:55 DEBUG "metric": { 20:37:55 DEBUG "pod": "ds-idrepo-0" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG [...] 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG 20:37:55 DEBUG "pod": "ds-idrepo-2" 20:37:55 DEBUG }, 20:37:55 DEBUG "value": [ 20:37:55 DEBUG 1713382675, 20:37:55 DEBUG "0" 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG ] 20:37:55 DEBUG } 20:37:55 DEBUG } 20:37:55 INFO 20:37:55 INFO [loop_until]: kubectl --namespace=xlou top pods 20:37:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:37:55 INFO [loop_until]: OK (rc = 0) 20:37:55 DEBUG --- stdout --- 20:37:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11844m 8890Mi am-b59c765b7-bgsxp 10887m 8713Mi am-b59c765b7-ch5q4 12133m 8850Mi ds-cts-0 12628m 7650Mi ds-cts-1 11591m 7555Mi ds-cts-2 12858m 7804Mi ds-idrepo-0 5215m 3084Mi ds-idrepo-1 5138m 2758Mi ds-idrepo-2 5144m 2682Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 9m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6737m 843Mi 20:37:55 DEBUG --- stderr --- 20:37:55 DEBUG 20:37:55 INFO 20:37:55 INFO [loop_until]: kubectl --namespace=xlou top node 20:37:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:37:55 INFO [loop_until]: OK (rc = 0) 20:37:55 DEBUG --- stdout --- 20:37:55 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 81m 0% 5499Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5087m 17% 5965Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11886m 39% 11421Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 2998Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12654m 42% 9844Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5003Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11857m 39% 11500Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27975m 93% 2908Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11665m 39% 10552Mi 9% gke-xlou-cdm-ds-0f601751-j488 5472m 18% 5202Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11582m 38% 10948Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5636m 18% 5803Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12917m 43% 9429Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6820m 42% 4631Mi 7% 20:37:55 DEBUG --- stderr --- 20:37:55 DEBUG 20:38:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382734" 20:38:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382734" 20:38:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382734" 20:38:54 INFO [http_cmd]: http status code OK 20:38:54 DEBUG --- status code --- 20:38:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:54 DEBUG --- http response --- 20:38:54 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:38:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382734" 20:38:54 INFO [http_cmd]: http status code OK 20:38:54 DEBUG --- status code --- 20:38:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:54 DEBUG --- http response --- 20:38:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:54 DEBUG ----- output ----- 20:38:54 DEBUG { 20:38:54 DEBUG "status": "success", 20:38:54 DEBUG "data": { 20:38:54 DEBUG "resultType": "vector", 20:38:54 DEBUG "result": [ 20:38:54 DEBUG { 20:38:54 DEBUG "metric": { 20:38:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:38:54 DEBUG }, 20:38:54 DEBUG "value": [ 20:38:54 DEBUG 20:38:54 DEBUG 20:38:54 DEBUG 20:38:54 DEBUG [...] 20:38:54 DEBUG 20:38:54 DEBUG 20:38:54 DEBUG 20:38:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:38:54 DEBUG }, 20:38:54 DEBUG "value": [ 20:38:54 DEBUG 1713382734, 20:38:54 DEBUG "1809.1333333333334" 20:38:54 DEBUG ] 20:38:54 DEBUG } 20:38:54 DEBUG ] 20:38:54 DEBUG } 20:38:54 DEBUG } 20:38:54 INFO [http_cmd]: http status code OK 20:38:54 DEBUG --- status code --- 20:38:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:54 DEBUG --- http response --- 20:38:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:54 DEBUG ----- output ----- 20:38:54 DEBUG { 20:38:54 DEBUG "status": "success", 20:38:54 DEBUG "data": { 20:38:54 DEBUG "resultType": "vector", 20:38:54 DEBUG "result": [ 20:38:54 DEBUG { 20:38:54 DEBUG "metric": { 20:38:54 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:38:54 DEBUG }, 20:38:54 DEBUG "value": [ 20:38:54 DEBUG 20:38:54 DEBUG 20:38:54 DEBUG 20:38:54 DEBUG [...] 20:38:54 DEBUG 20:38:54 DEBUG 20:38:54 DEBUG 20:38:54 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:38:54 DEBUG }, 20:38:54 DEBUG "value": [ 20:38:54 DEBUG 1713382734, 20:38:54 DEBUG "986316800" 20:38:54 DEBUG ] 20:38:54 DEBUG } 20:38:54 DEBUG ] 20:38:54 DEBUG } 20:38:54 DEBUG } 20:38:54 INFO [http_cmd]: http status code OK 20:38:54 DEBUG --- status code --- 20:38:54 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:54 DEBUG --- http response --- 20:38:54 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:54 DEBUG ----- output ----- 20:38:54 DEBUG { 20:38:54 DEBUG "status": "success", 20:38:54 DEBUG "data": { 20:38:54 DEBUG "resultType": "vector", 20:38:54 DEBUG "result": [ 20:38:54 DEBUG { 20:38:54 DEBUG "metric": { 20:38:54 DEBUG "pod": "am-b59c765b7-ch5q4" 20:38:54 DEBUG }, 20:38:54 DEBUG "value": [ 20:38:54 DEBUG 20:38:54 DEBUG 20:38:54 DEBUG 20:38:54 DEBUG [...] 20:38:54 DEBUG 20:38:54 DEBUG 20:38:54 DEBUG 20:38:54 DEBUG "pod": "am-b59c765b7-bgsxp" 20:38:54 DEBUG }, 20:38:54 DEBUG "value": [ 20:38:54 DEBUG 1713382734, 20:38:54 DEBUG "0.007" 20:38:54 DEBUG ] 20:38:54 DEBUG } 20:38:54 DEBUG ] 20:38:54 DEBUG } 20:38:54 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382734" 20:38:54 DEBUG } 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382734, 20:38:55 DEBUG "1809.2666666666667" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382735" 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382735" 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "data": { 20:38:55 DEBUG "pod": "ds-idrepo-0" 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG }, 20:38:55 DEBUG "result": [ 20:38:55 DEBUG "value": [ 20:38:55 DEBUG { 20:38:55 DEBUG 20:38:55 DEBUG "metric": { 20:38:55 DEBUG 20:38:55 DEBUG "instance": "10.142.0.107:9100" 20:38:55 DEBUG 20:38:55 DEBUG }, 20:38:55 DEBUG [...] 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "ds-cts-2" 20:38:55 DEBUG [...] 20:38:55 DEBUG }, 20:38:55 DEBUG 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG 20:38:55 DEBUG "0" 20:38:55 DEBUG "instance": "10.142.0.43:9100" 20:38:55 DEBUG ] 20:38:55 DEBUG }, 20:38:55 DEBUG } 20:38:55 DEBUG "value": [ 20:38:55 DEBUG ] 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG } 20:38:55 DEBUG "60644457.93333334" 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 ERROR [http_cmd]: ERROR 20:38:55 ERROR --- status code --- 20:38:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:38:55 ERROR --- http response --- 20:38:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:38:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382735" 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG { 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "0" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "instance": "10.142.0.107:9100" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "instance": "10.142.0.43:9100" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "0.16799999999999973" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "pod": "ds-cts-0" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "ds-cts-2" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "0.00007064000000000001" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "1817.5333333333333" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "31363494.73925239" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "11.521162450206347" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382735" 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "instance": "10.142.0.107:9100" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "instance": "10.142.0.43:9100" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "60636612.36666667" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382735" 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382735" 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:38:55 DEBUG "instance": "10.142.0.107:9100" 20:38:55 DEBUG }, 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:38:55 DEBUG "instance": "10.142.0.43:9100" 20:38:55 DEBUG }, 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "1808.6333333333332" 20:38:55 DEBUG "110225912832" 20:38:55 DEBUG ] 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "6.655764318746529" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "pod": "ds-idrepo-0" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "ds-idrepo-2" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "0" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG "data": { 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG "result": [ 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:38:55 DEBUG "pod": "ds-idrepo-0" 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG }, 20:38:55 DEBUG { 20:38:55 DEBUG "value": [ 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG 20:38:55 DEBUG "data": { 20:38:55 DEBUG 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG 20:38:55 DEBUG "result": [ 20:38:55 DEBUG [...] 20:38:55 DEBUG { 20:38:55 DEBUG 20:38:55 DEBUG "metric": { 20:38:55 DEBUG 20:38:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:38:55 DEBUG 20:38:55 DEBUG }, 20:38:55 DEBUG "pod": "ds-idrepo-2" 20:38:55 DEBUG "value": [ 20:38:55 DEBUG }, 20:38:55 DEBUG 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG 20:38:55 DEBUG "0" 20:38:55 DEBUG [...] 20:38:55 DEBUG ] 20:38:55 DEBUG 20:38:55 DEBUG } 20:38:55 DEBUG 20:38:55 DEBUG ] 20:38:55 DEBUG 20:38:55 DEBUG } 20:38:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:38:55 DEBUG } 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "0" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "328530" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382735" 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "0" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "pod": "ds-cts-0" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "ds-cts-2" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "29.966666666666665" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "pod": "ds-cts-0" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "ds-cts-2" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "0" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "instance": "10.142.0.107:9100" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "instance": "10.142.0.43:9100" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "0.08066666666666678" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382735" 20:38:55 INFO [http_cmd]: http status code OK 20:38:55 DEBUG --- status code --- 20:38:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:55 DEBUG --- http response --- 20:38:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:55 DEBUG ----- output ----- 20:38:55 DEBUG { 20:38:55 DEBUG "status": "success", 20:38:55 DEBUG "data": { 20:38:55 DEBUG "resultType": "vector", 20:38:55 DEBUG "result": [ 20:38:55 DEBUG { 20:38:55 DEBUG "metric": { 20:38:55 DEBUG "pod": "ds-idrepo-0" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG [...] 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG 20:38:55 DEBUG "pod": "ds-idrepo-2" 20:38:55 DEBUG }, 20:38:55 DEBUG "value": [ 20:38:55 DEBUG 1713382735, 20:38:55 DEBUG "0" 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG ] 20:38:55 DEBUG } 20:38:55 DEBUG } 20:38:55 INFO 20:38:55 INFO [loop_until]: kubectl --namespace=xlou top pods 20:38:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:38:55 INFO [loop_until]: OK (rc = 0) 20:38:55 DEBUG --- stdout --- 20:38:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11670m 8892Mi am-b59c765b7-bgsxp 10711m 8714Mi am-b59c765b7-ch5q4 11698m 8848Mi ds-cts-0 12763m 7654Mi ds-cts-1 10981m 7551Mi ds-cts-2 12920m 7833Mi ds-idrepo-0 5160m 3085Mi ds-idrepo-1 5324m 2758Mi ds-idrepo-2 5111m 2682Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 5m 1003Mi lodemon-db4967764-9zt9t 9m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6621m 845Mi 20:38:55 DEBUG --- stderr --- 20:38:55 DEBUG 20:38:55 INFO 20:38:55 INFO [loop_until]: kubectl --namespace=xlou top node 20:38:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:38:56 INFO [loop_until]: OK (rc = 0) 20:38:56 DEBUG --- stdout --- 20:38:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 79m 0% 5501Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5156m 17% 5973Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11805m 39% 11416Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 2990Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12580m 42% 9836Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 170m 0% 5004Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11857m 39% 11492Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28080m 94% 2916Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11652m 39% 10552Mi 9% gke-xlou-cdm-ds-0f601751-j488 5409m 18% 5208Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11383m 38% 10955Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5463m 18% 5802Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12876m 43% 9450Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6805m 42% 4630Mi 7% 20:38:56 DEBUG --- stderr --- 20:38:56 DEBUG 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "1835.8666666666666" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382795" 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "1835.6" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "989433856" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "0.007" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382795" 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "ds-cts-0" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "ds-cts-2" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "0.00036887333333333335" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "instance": "10.142.0.107:9100" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "instance": "10.142.0.43:9100" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "0.08469999999999989" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "instance": "10.142.0.107:9100" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "instance": "10.142.0.43:9100" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "60846962.16666667" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 ERROR [http_cmd]: ERROR 20:39:55 DEBUG --- status code --- 20:39:55 ERROR --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 ERROR --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "ds-idrepo-0" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "ds-cts-2" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "0" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:39:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "0" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "1830.9666666666667" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "3.4063425593903496" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "25931687.543273438" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382795" 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "6.651500675755899" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG "metric": { 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG "pod": "ds-idrepo-0" 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG }, 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG "value": [ 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:55 DEBUG 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG 20:39:55 DEBUG { 20:39:55 DEBUG 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG [...] 20:39:55 DEBUG "data": { 20:39:55 DEBUG 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG 20:39:55 DEBUG "result": [ 20:39:55 DEBUG 20:39:55 DEBUG { 20:39:55 DEBUG "pod": "ds-idrepo-2" 20:39:55 DEBUG "metric": { 20:39:55 DEBUG }, 20:39:55 DEBUG "instance": "10.142.0.107:9100" 20:39:55 DEBUG "value": [ 20:39:55 DEBUG }, 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG "0" 20:39:55 DEBUG 20:39:55 DEBUG ] 20:39:55 DEBUG 20:39:55 DEBUG } 20:39:55 DEBUG 20:39:55 DEBUG ] 20:39:55 DEBUG [...] 20:39:55 DEBUG } 20:39:55 DEBUG 20:39:55 DEBUG } 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "instance": "10.142.0.43:9100" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "60839715.166666664" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "metric": { 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:39:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG }, 20:39:55 DEBUG }, 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG "value": [ 20:39:55 DEBUG "value": [ 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG { 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "data": { 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG 20:39:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:39:55 DEBUG "result": [ 20:39:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:39:55 DEBUG }, 20:39:55 DEBUG { 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "instance": "10.142.0.107:9100" 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "0" 20:39:55 DEBUG }, 20:39:55 DEBUG "1836.2333333333333" 20:39:55 DEBUG ] 20:39:55 DEBUG "value": [ 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 DEBUG [...] 20:39:55 DEBUG } 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "instance": "10.142.0.43:9100" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "113173283840" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "ds-idrepo-0" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "ds-idrepo-2" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "0" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "ds-cts-0" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "ds-cts-2" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "0" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "instance": "10.142.0.107:9100" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "instance": "10.142.0.43:9100" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "0.05466666666666665" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "0" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "328892" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382795" 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "ds-idrepo-0" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "ds-idrepo-2" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "0" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO [http_cmd]: http status code OK 20:39:55 DEBUG --- status code --- 20:39:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:55 DEBUG --- http response --- 20:39:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:39:55 DEBUG ----- output ----- 20:39:55 DEBUG { 20:39:55 DEBUG "status": "success", 20:39:55 DEBUG "data": { 20:39:55 DEBUG "resultType": "vector", 20:39:55 DEBUG "result": [ 20:39:55 DEBUG { 20:39:55 DEBUG "metric": { 20:39:55 DEBUG "pod": "ds-cts-0" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG [...] 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG 20:39:55 DEBUG "pod": "ds-cts-2" 20:39:55 DEBUG }, 20:39:55 DEBUG "value": [ 20:39:55 DEBUG 1713382795, 20:39:55 DEBUG "30" 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG ] 20:39:55 DEBUG } 20:39:55 DEBUG } 20:39:55 INFO 20:39:55 INFO [loop_until]: kubectl --namespace=xlou top pods 20:39:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:39:55 INFO [loop_until]: OK (rc = 0) 20:39:55 DEBUG --- stdout --- 20:39:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11924m 8892Mi am-b59c765b7-bgsxp 11136m 8712Mi am-b59c765b7-ch5q4 11755m 8849Mi ds-cts-0 12392m 7658Mi ds-cts-1 11564m 7535Mi ds-cts-2 12712m 7837Mi ds-idrepo-0 5170m 3085Mi ds-idrepo-1 5112m 2758Mi ds-idrepo-2 5174m 2682Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 5m 1003Mi lodemon-db4967764-9zt9t 9m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6470m 846Mi 20:39:55 DEBUG --- stderr --- 20:39:55 DEBUG 20:39:56 INFO 20:39:56 INFO [loop_until]: kubectl --namespace=xlou top node 20:39:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:39:56 INFO [loop_until]: OK (rc = 0) 20:39:56 DEBUG --- stdout --- 20:39:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 81m 0% 5497Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5090m 17% 5966Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11673m 39% 11430Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 2995Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12493m 41% 9850Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5007Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11852m 39% 11496Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27119m 90% 2926Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11850m 39% 10556Mi 9% gke-xlou-cdm-ds-0f601751-j488 5356m 17% 5211Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11656m 39% 10943Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5603m 18% 5818Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12916m 43% 9452Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6570m 41% 4629Mi 7% 20:39:56 DEBUG --- stderr --- 20:39:56 DEBUG 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "1829.5333333333333" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "0.007" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "1829.5666666666666" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382855" 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382855" 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "988844032" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382855" 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "ds-idrepo-0" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "ds-cts-2" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "0" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "instance": "10.142.0.107:9100" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "instance": "10.142.0.43:9100" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "0.06206666666666649" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 ERROR [http_cmd]: ERROR 20:40:55 ERROR --- status code --- 20:40:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:40:55 ERROR --- http response --- 20:40:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:40:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "ds-cts-0" 20:40:55 DEBUG "instance": "10.142.0.107:9100" 20:40:55 DEBUG }, 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "ds-cts-2" 20:40:55 DEBUG "instance": "10.142.0.43:9100" 20:40:55 DEBUG }, 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "0" 20:40:55 DEBUG "60858395.2" 20:40:55 DEBUG ] 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "1846.1333333333332" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "0" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "0" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "0" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "6.686940868784371" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "ds-idrepo-0" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "ds-idrepo-2" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "0" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "1829.5333333333333" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "instance": "10.142.0.107:9100" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "instance": "10.142.0.43:9100" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "60851106.2" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382855" 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "instance": "10.142.0.107:9100" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "instance": "10.142.0.43:9100" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "116077640704" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "ds-idrepo-0" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "ds-idrepo-2" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "0" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "0" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "ds-cts-0" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "ds-cts-2" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "0" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382855" 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "ds-cts-0" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "ds-cts-2" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "29.966666666666665" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "instance": "10.142.0.107:9100" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "instance": "10.142.0.43:9100" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "0.04266666666666657" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "idm-b85f9d978-wnhxb" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "981.4387191713847" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "329028" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382855" 20:40:55 INFO [http_cmd]: http status code OK 20:40:55 DEBUG --- status code --- 20:40:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:55 DEBUG --- http response --- 20:40:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:55 DEBUG ----- output ----- 20:40:55 DEBUG { 20:40:55 DEBUG "status": "success", 20:40:55 DEBUG "data": { 20:40:55 DEBUG "resultType": "vector", 20:40:55 DEBUG "result": [ 20:40:55 DEBUG { 20:40:55 DEBUG "metric": { 20:40:55 DEBUG "pod": "ds-idrepo-0" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG [...] 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG 20:40:55 DEBUG "pod": "ds-idrepo-2" 20:40:55 DEBUG }, 20:40:55 DEBUG "value": [ 20:40:55 DEBUG 1713382855, 20:40:55 DEBUG "0" 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG ] 20:40:55 DEBUG } 20:40:55 DEBUG } 20:40:55 INFO 20:40:55 INFO [loop_until]: kubectl --namespace=xlou top pods 20:40:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:40:55 INFO [loop_until]: OK (rc = 0) 20:40:55 DEBUG --- stdout --- 20:40:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12059m 8893Mi am-b59c765b7-bgsxp 10855m 8717Mi am-b59c765b7-ch5q4 11906m 8850Mi ds-cts-0 12863m 7665Mi ds-cts-1 11381m 7560Mi ds-cts-2 12842m 7812Mi ds-idrepo-0 5209m 3085Mi ds-idrepo-1 5179m 2758Mi ds-idrepo-2 5195m 2682Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 11m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6567m 850Mi 20:40:55 DEBUG --- stderr --- 20:40:55 DEBUG 20:40:56 INFO 20:40:56 INFO [loop_until]: kubectl --namespace=xlou top node 20:40:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:40:56 INFO [loop_until]: OK (rc = 0) 20:40:56 DEBUG --- stdout --- 20:40:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 81m 0% 5498Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5079m 17% 5969Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11746m 39% 11429Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 2992Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12540m 42% 9856Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5025Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11976m 40% 11504Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28181m 94% 2933Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11754m 39% 10551Mi 9% gke-xlou-cdm-ds-0f601751-j488 5230m 17% 5212Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11443m 38% 10955Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5657m 18% 5819Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12774m 42% 9443Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6831m 42% 4629Mi 7% 20:40:56 DEBUG --- stderr --- 20:40:56 DEBUG 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "1815.5" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "0.007" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "1028321280" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "1815.6" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382915" 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "instance": "10.142.0.107:9100" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "instance": "10.142.0.43:9100" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "60539629.46666667" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "instance": "10.142.0.107:9100" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "instance": "10.142.0.43:9100" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "0.09843333333333343" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382915" 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:41:55 DEBUG "pod": "ds-idrepo-0" 20:41:55 DEBUG }, 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:41:55 DEBUG "pod": "ds-cts-2" 20:41:55 DEBUG }, 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "1807.6" 20:41:55 DEBUG "0" 20:41:55 DEBUG ] 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 ERROR [http_cmd]: ERROR 20:41:55 ERROR --- status code --- 20:41:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:41:55 ERROR --- http response --- 20:41:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:41:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "ds-cts-0" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "ds-cts-2" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "0.0001666666666666667" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "0" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "0" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "0" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "ds-idrepo-0" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "ds-idrepo-2" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "0" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "6.698469661825229" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382915" 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:41:55 DEBUG { 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG { 20:41:55 DEBUG "data": { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "result": [ 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG { 20:41:55 DEBUG "result": [ 20:41:55 DEBUG "metric": { 20:41:55 DEBUG { 20:41:55 DEBUG "pod": "ds-idrepo-0" 20:41:55 DEBUG "metric": { 20:41:55 DEBUG }, 20:41:55 DEBUG "pod": "ds-cts-0" 20:41:55 DEBUG "value": [ 20:41:55 DEBUG }, 20:41:55 DEBUG 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "ds-idrepo-2" 20:41:55 DEBUG 20:41:55 DEBUG }, 20:41:55 DEBUG "pod": "ds-cts-2" 20:41:55 DEBUG "value": [ 20:41:55 DEBUG }, 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG "0" 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG ] 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG "0" 20:41:55 DEBUG } 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG ] 20:41:55 DEBUG ] 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG } 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "1815.6" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "instance": "10.142.0.107:9100" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "instance": "10.142.0.43:9100" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "60532023.1" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382915" 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "ds-idrepo-0" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "0" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "328592" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "instance": "10.142.0.107:9100" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "instance": "10.142.0.43:9100" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "118834297856" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "ds-idrepo-0" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "ds-idrepo-2" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "0" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382915" 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382915" 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "ds-cts-0" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "ds-cts-2" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "29.933333333333334" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "instance": "10.142.0.107:9100" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "instance": "10.142.0.43:9100" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "0.05100000000000004" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO [http_cmd]: http status code OK 20:41:55 DEBUG --- status code --- 20:41:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:41:55 DEBUG --- http response --- 20:41:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:41:55 DEBUG ----- output ----- 20:41:55 DEBUG { 20:41:55 DEBUG "status": "success", 20:41:55 DEBUG "data": { 20:41:55 DEBUG "resultType": "vector", 20:41:55 DEBUG "result": [ 20:41:55 DEBUG { 20:41:55 DEBUG "metric": { 20:41:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG [...] 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG 20:41:55 DEBUG "pod": "ds-idrepo-0" 20:41:55 DEBUG }, 20:41:55 DEBUG "value": [ 20:41:55 DEBUG 1713382915, 20:41:55 DEBUG "3863.156992197548" 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG ] 20:41:55 DEBUG } 20:41:55 DEBUG } 20:41:55 INFO 20:41:55 INFO [loop_until]: kubectl --namespace=xlou top pods 20:41:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:55 INFO [loop_until]: OK (rc = 0) 20:41:55 DEBUG --- stdout --- 20:41:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12196m 8892Mi am-b59c765b7-bgsxp 11005m 8718Mi am-b59c765b7-ch5q4 11747m 8848Mi ds-cts-0 12496m 7675Mi ds-cts-1 11769m 7543Mi ds-cts-2 12766m 7818Mi ds-idrepo-0 5259m 3086Mi ds-idrepo-1 5182m 2758Mi ds-idrepo-2 5160m 2682Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 10m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 9m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6595m 888Mi 20:41:55 DEBUG --- stderr --- 20:41:55 DEBUG 20:41:56 INFO 20:41:56 INFO [loop_until]: kubectl --namespace=xlou top node 20:41:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:41:56 INFO [loop_until]: OK (rc = 0) 20:41:56 DEBUG --- stdout --- 20:41:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 5495Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5110m 17% 5968Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11705m 39% 11417Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 2997Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12485m 41% 9840Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5008Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11565m 38% 11487Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27993m 93% 2929Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11737m 39% 10558Mi 9% gke-xlou-cdm-ds-0f601751-j488 5445m 18% 5219Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11460m 38% 10963Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5652m 18% 5809Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13283m 44% 9453Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6894m 43% 4667Mi 7% 20:41:56 DEBUG --- stderr --- 20:41:56 DEBUG 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713382975" 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "0.007" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "1824.6666666666667" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "1032327168" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "ds-idrepo-0" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "ds-cts-2" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "0" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "1824.4333333333334" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "0" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382975" 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "instance": "10.142.0.107:9100" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "instance": "10.142.0.43:9100" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "60756006.8" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "instance": "10.142.0.107:9100" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "instance": "10.142.0.43:9100" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "0.1508333333333335" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713382975" 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "1837.2666666666667" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 ERROR [http_cmd]: ERROR 20:42:55 ERROR --- status code --- 20:42:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:42:55 ERROR --- http response --- 20:42:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:42:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "ds-cts-0" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "ds-cts-2" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "0.0006778133333333334" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "11.152862140249578" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "33528132.034610942" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "ds-cts-0" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "ds-cts-2" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "0" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713382975" 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "ds-idrepo-0" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "ds-idrepo-2" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "0" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "0" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "328500" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "6.64454287083019" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "1824.6" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382975" 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "ds-idrepo-0" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "ds-idrepo-2" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "0" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "instance": "10.142.0.107:9100" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "instance": "10.142.0.43:9100" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "60747762.46666667" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "instance": "10.142.0.107:9100" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "instance": "10.142.0.43:9100" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "121387996160" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "0" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "ds-idrepo-0" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "ds-idrepo-2" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "0" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713382975" 20:42:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713382975" 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "pod": "ds-cts-0" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "pod": "ds-cts-2" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "30.03333333333333" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO [http_cmd]: http status code OK 20:42:55 DEBUG --- status code --- 20:42:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:42:55 DEBUG --- http response --- 20:42:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:42:55 DEBUG ----- output ----- 20:42:55 DEBUG { 20:42:55 DEBUG "status": "success", 20:42:55 DEBUG "data": { 20:42:55 DEBUG "resultType": "vector", 20:42:55 DEBUG "result": [ 20:42:55 DEBUG { 20:42:55 DEBUG "metric": { 20:42:55 DEBUG "instance": "10.142.0.107:9100" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG [...] 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG 20:42:55 DEBUG "instance": "10.142.0.43:9100" 20:42:55 DEBUG }, 20:42:55 DEBUG "value": [ 20:42:55 DEBUG 1713382975, 20:42:55 DEBUG "0.07400000000000015" 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG ] 20:42:55 DEBUG } 20:42:55 DEBUG } 20:42:55 INFO 20:42:55 INFO [loop_until]: kubectl --namespace=xlou top pods 20:42:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:42:55 INFO [loop_until]: OK (rc = 0) 20:42:55 DEBUG --- stdout --- 20:42:55 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11697m 8892Mi am-b59c765b7-bgsxp 11043m 8715Mi am-b59c765b7-ch5q4 11618m 8848Mi ds-cts-0 12350m 7695Mi ds-cts-1 11582m 7545Mi ds-cts-2 12542m 7831Mi ds-idrepo-0 5294m 3086Mi ds-idrepo-1 5170m 2758Mi ds-idrepo-2 5134m 2723Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 965Mi idm-b85f9d978-wnhxb 6m 1003Mi lodemon-db4967764-9zt9t 9m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6473m 891Mi 20:42:55 DEBUG --- stderr --- 20:42:55 DEBUG 20:42:56 INFO 20:42:56 INFO [loop_until]: kubectl --namespace=xlou top node 20:42:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:42:56 INFO [loop_until]: OK (rc = 0) 20:42:56 DEBUG --- stdout --- 20:42:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 80m 0% 5485Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5139m 17% 5962Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11707m 39% 11427Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 2995Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12523m 41% 9857Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5012Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11960m 40% 11494Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27861m 93% 2911Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11765m 39% 10547Mi 9% gke-xlou-cdm-ds-0f601751-j488 5416m 18% 5249Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11577m 38% 10957Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5667m 18% 5810Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12452m 41% 9468Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6788m 42% 4668Mi 7% 20:42:56 DEBUG --- stderr --- 20:42:56 DEBUG 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:43:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:43:55 DEBUG }, 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:43:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:43:55 DEBUG }, 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "0.007" 20:43:55 DEBUG "1828.8" 20:43:55 DEBUG ] 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383035" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "ds-idrepo-0" 20:43:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:43:55 DEBUG }, 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "ds-cts-2" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:43:55 DEBUG }, 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "0" 20:43:55 DEBUG "1828.8" 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:55 DEBUG ] 20:43:55 DEBUG ] 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 DEBUG { 20:43:55 DEBUG ] 20:43:55 DEBUG ] 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 DEBUG "data": { 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "1034330112" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383035" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "0" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "instance": "10.142.0.107:9100" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "instance": "10.142.0.43:9100" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "0.15556666666666671" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 ERROR [http_cmd]: ERROR 20:43:55 ERROR --- status code --- 20:43:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:43:55 ERROR --- http response --- 20:43:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:43:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "instance": "10.142.0.107:9100" 20:43:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:43:55 DEBUG }, 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "instance": "10.142.0.43:9100" 20:43:55 DEBUG "pod": "ds-cts-0" 20:43:55 DEBUG }, 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "60758633.46666667" 20:43:55 DEBUG "12.443559617121206" 20:43:55 DEBUG ] 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "ds-cts-0" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "ds-cts-2" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "0.000033333333333333335" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "1829.4666666666667" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "ds-cts-0" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "290641862.6693155" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "328006" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "ds-cts-0" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "36125432.83366444" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383035" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "ds-cts-0" 20:43:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:43:55 DEBUG }, 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "ds-cts-2" 20:43:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:43:55 DEBUG }, 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "0" 20:43:55 DEBUG "1828.6" 20:43:55 DEBUG ] 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "6.6075662089182785" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383035" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "ds-idrepo-0" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "ds-idrepo-2" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "0" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "instance": "10.142.0.107:9100" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "instance": "10.142.0.43:9100" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "124246463488" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383035" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "ds-idrepo-0" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "ds-idrepo-2" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "0" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "instance": "10.142.0.107:9100" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "instance": "10.142.0.43:9100" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "60751542.4" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "ds-cts-0" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "40153227.81289507" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "ds-idrepo-0" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "ds-idrepo-2" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "0" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383035" 20:43:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383035" 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "pod": "ds-cts-0" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "pod": "ds-cts-2" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "29.966666666666665" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:55 INFO [http_cmd]: http status code OK 20:43:55 DEBUG --- status code --- 20:43:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:43:55 DEBUG --- http response --- 20:43:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:43:55 DEBUG ----- output ----- 20:43:55 DEBUG { 20:43:55 DEBUG "status": "success", 20:43:55 DEBUG "data": { 20:43:55 DEBUG "resultType": "vector", 20:43:55 DEBUG "result": [ 20:43:55 DEBUG { 20:43:55 DEBUG "metric": { 20:43:55 DEBUG "instance": "10.142.0.107:9100" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG [...] 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG 20:43:55 DEBUG "instance": "10.142.0.43:9100" 20:43:55 DEBUG }, 20:43:55 DEBUG "value": [ 20:43:55 DEBUG 1713383035, 20:43:55 DEBUG "0.07533333333333336" 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG ] 20:43:55 DEBUG } 20:43:55 DEBUG } 20:43:56 INFO 20:43:56 INFO [loop_until]: kubectl --namespace=xlou top pods 20:43:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:43:56 INFO [loop_until]: OK (rc = 0) 20:43:56 DEBUG --- stdout --- 20:43:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12176m 8890Mi am-b59c765b7-bgsxp 10945m 8716Mi am-b59c765b7-ch5q4 11432m 8851Mi ds-cts-0 12136m 7695Mi ds-cts-1 11431m 7554Mi ds-cts-2 13746m 7830Mi ds-idrepo-0 5216m 3085Mi ds-idrepo-1 5076m 2758Mi ds-idrepo-2 5175m 2723Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 965Mi idm-b85f9d978-wnhxb 5m 1003Mi lodemon-db4967764-9zt9t 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6648m 892Mi 20:43:56 DEBUG --- stderr --- 20:43:56 DEBUG 20:43:56 INFO 20:43:56 INFO [loop_until]: kubectl --namespace=xlou top node 20:43:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:43:56 INFO [loop_until]: OK (rc = 0) 20:43:56 DEBUG --- stdout --- 20:43:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 81m 0% 5485Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5076m 17% 5967Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11651m 39% 11433Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 2987Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12268m 41% 9867Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5009Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12006m 40% 11506Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27769m 93% 2920Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11586m 38% 10552Mi 9% gke-xlou-cdm-ds-0f601751-j488 5381m 18% 5265Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11195m 37% 10985Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5663m 18% 5821Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12963m 43% 9471Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6560m 41% 4665Mi 7% 20:43:56 DEBUG --- stderr --- 20:43:56 DEBUG 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "0.008" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "1819.5666666666666" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383095" 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "1819.1" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "1036419072" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "ds-idrepo-0" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "ds-cts-2" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "0" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383095" 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383095" 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "instance": "10.142.0.107:9100" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "instance": "10.142.0.43:9100" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "60898729.86666666" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 ERROR [http_cmd]: ERROR 20:44:55 ERROR --- status code --- 20:44:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:44:55 ERROR --- http response --- 20:44:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:44:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383095" 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG { 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG { 20:44:55 DEBUG "data": { 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG "data": { 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "data": { 20:44:55 DEBUG { 20:44:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "metric": { 20:44:55 DEBUG }, 20:44:55 DEBUG "result": [ 20:44:55 DEBUG "pod": "ds-cts-0" 20:44:55 DEBUG "value": [ 20:44:55 DEBUG { 20:44:55 DEBUG }, 20:44:55 DEBUG 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG }, 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG }, 20:44:55 DEBUG 20:44:55 DEBUG "pod": "ds-cts-2" 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG }, 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG 20:44:55 DEBUG "value": [ 20:44:55 DEBUG "0" 20:44:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG ] 20:44:55 DEBUG }, 20:44:55 DEBUG "0.00022858666666666664" 20:44:55 DEBUG } 20:44:55 DEBUG "value": [ 20:44:55 DEBUG ] 20:44:55 DEBUG ] 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 DEBUG "1814.8666666666666" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "instance": "10.142.0.107:9100" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "instance": "10.142.0.43:9100" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "0.15386666666666618" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "0.00003030855199222712" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "0" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "0" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "1820.7666666666667" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383095" 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "ds-idrepo-0" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "ds-idrepo-2" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "0" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "328254" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "ds-cts-0" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "ds-cts-2" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "0" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383095" 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "6.6228108164484185" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "instance": "10.142.0.107:9100" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "instance": "10.142.0.43:9100" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "60891133.63333333" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383095" 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "instance": "10.142.0.107:9100" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "instance": "10.142.0.43:9100" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "127108711424" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "ds-idrepo-0" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "ds-idrepo-2" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "0" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "ds-idrepo-0" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "ds-idrepo-2" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "0" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383095" 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383095" 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "0" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "pod": "ds-cts-0" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "pod": "ds-cts-2" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "29.933333333333334" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:55 INFO [http_cmd]: http status code OK 20:44:55 DEBUG --- status code --- 20:44:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:44:55 DEBUG --- http response --- 20:44:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:44:55 DEBUG ----- output ----- 20:44:55 DEBUG { 20:44:55 DEBUG "status": "success", 20:44:55 DEBUG "data": { 20:44:55 DEBUG "resultType": "vector", 20:44:55 DEBUG "result": [ 20:44:55 DEBUG { 20:44:55 DEBUG "metric": { 20:44:55 DEBUG "instance": "10.142.0.107:9100" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG [...] 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG 20:44:55 DEBUG "instance": "10.142.0.43:9100" 20:44:55 DEBUG }, 20:44:55 DEBUG "value": [ 20:44:55 DEBUG 1713383095, 20:44:55 DEBUG "0.07966666666666683" 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG ] 20:44:55 DEBUG } 20:44:55 DEBUG } 20:44:56 INFO 20:44:56 INFO [loop_until]: kubectl --namespace=xlou top pods 20:44:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:44:56 INFO [loop_until]: OK (rc = 0) 20:44:56 DEBUG --- stdout --- 20:44:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11935m 8891Mi am-b59c765b7-bgsxp 10853m 8717Mi am-b59c765b7-ch5q4 11964m 8850Mi ds-cts-0 12251m 7683Mi ds-cts-1 11737m 7554Mi ds-cts-2 13319m 7827Mi ds-idrepo-0 5128m 3085Mi ds-idrepo-1 5237m 2758Mi ds-idrepo-2 5101m 2722Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 10m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6548m 893Mi 20:44:56 DEBUG --- stderr --- 20:44:56 DEBUG 20:44:56 INFO 20:44:56 INFO [loop_until]: kubectl --namespace=xlou top node 20:44:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:44:56 INFO [loop_until]: OK (rc = 0) 20:44:56 DEBUG --- stdout --- 20:44:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 82m 0% 5490Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5135m 17% 5957Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11766m 39% 11436Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 2994Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12450m 41% 9861Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5011Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11966m 40% 11501Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27099m 90% 2913Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11733m 39% 10555Mi 9% gke-xlou-cdm-ds-0f601751-j488 5362m 17% 5251Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11793m 39% 10981Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5645m 18% 5819Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13252m 44% 9488Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6782m 42% 4669Mi 7% 20:44:56 DEBUG --- stderr --- 20:44:56 DEBUG 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383155" 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "1832.1" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "1831.0333333333333" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "1037901824" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "0.0095" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "ds-idrepo-0" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "ds-cts-2" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "0" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383155" 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383155" 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:45:55 DEBUG "pod": "ds-cts-0" 20:45:55 DEBUG }, 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:45:55 DEBUG "pod": "ds-cts-2" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG }, 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "value": [ 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG "1827.1666666666667" 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG ] 20:45:55 DEBUG "0" 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG } 20:45:55 DEBUG { 20:45:55 DEBUG } 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "instance": "10.142.0.107:9100" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "instance": "10.142.0.43:9100" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "0.052366666666666596" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383155" 20:45:55 ERROR [http_cmd]: ERROR 20:45:55 ERROR --- status code --- 20:45:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:45:55 ERROR --- http response --- 20:45:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:45:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "instance": "10.142.0.107:9100" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "instance": "10.142.0.43:9100" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "60850084.03333333" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "0" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "ds-idrepo-1" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "0.000027712418300646062" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "ds-idrepo-0" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "ds-idrepo-2" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "0" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "ds-idrepo-1" 20:45:55 DEBUG "pod": "ds-idrepo-1" 20:45:55 DEBUG }, 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:45:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:45:55 DEBUG }, 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "0" 20:45:55 DEBUG "0" 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG ] 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG ] 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG } 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG } 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG ] 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG ] 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG } 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:45:55 DEBUG } 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "ds-cts-0" 20:45:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:45:55 DEBUG }, 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "ds-cts-2" 20:45:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:45:55 DEBUG }, 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "0" 20:45:55 DEBUG "1832.8" 20:45:55 DEBUG ] 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383155" 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "328035.5" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383155" 20:45:55 DEBUG "6.6095476207184065" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "instance": "10.142.0.107:9100" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "instance": "10.142.0.43:9100" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "60842797.46666667" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383155" 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "instance": "10.142.0.107:9100" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "instance": "10.142.0.43:9100" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "129585179648" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "ds-idrepo-0" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "ds-idrepo-2" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "0" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383155" 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "ds-cts-0" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "ds-cts-2" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "30" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383155" 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "instance": "10.142.0.107:9100" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "instance": "10.142.0.43:9100" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "0.035333333333333404" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "ds-idrepo-0" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "ds-idrepo-2" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "0" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:55 INFO [http_cmd]: http status code OK 20:45:55 DEBUG --- status code --- 20:45:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:45:55 DEBUG --- http response --- 20:45:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:45:55 DEBUG ----- output ----- 20:45:55 DEBUG { 20:45:55 DEBUG "status": "success", 20:45:55 DEBUG "data": { 20:45:55 DEBUG "resultType": "vector", 20:45:55 DEBUG "result": [ 20:45:55 DEBUG { 20:45:55 DEBUG "metric": { 20:45:55 DEBUG "pod": "ds-idrepo-1" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG [...] 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG 20:45:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:45:55 DEBUG }, 20:45:55 DEBUG "value": [ 20:45:55 DEBUG 1713383155, 20:45:55 DEBUG "0" 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG ] 20:45:55 DEBUG } 20:45:55 DEBUG } 20:45:56 INFO 20:45:56 INFO [loop_until]: kubectl --namespace=xlou top pods 20:45:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:45:56 INFO [loop_until]: OK (rc = 0) 20:45:56 DEBUG --- stdout --- 20:45:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11588m 8891Mi am-b59c765b7-bgsxp 10677m 8717Mi am-b59c765b7-ch5q4 10991m 8849Mi ds-cts-0 12654m 7680Mi ds-cts-1 11468m 7556Mi ds-cts-2 12843m 7830Mi ds-idrepo-0 5152m 3082Mi ds-idrepo-1 5100m 2758Mi ds-idrepo-2 5046m 2722Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 10m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6432m 895Mi 20:45:56 DEBUG --- stderr --- 20:45:56 DEBUG 20:45:56 INFO 20:45:56 INFO [loop_until]: kubectl --namespace=xlou top node 20:45:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:45:56 INFO [loop_until]: OK (rc = 0) 20:45:56 DEBUG --- stdout --- 20:45:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 81m 0% 5479Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5126m 17% 5971Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11740m 39% 11433Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 2991Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12491m 41% 9878Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5028Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11910m 39% 11500Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27669m 92% 2910Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11627m 38% 10547Mi 9% gke-xlou-cdm-ds-0f601751-j488 5319m 17% 5259Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11485m 38% 10984Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5625m 18% 5824Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12852m 43% 9489Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6720m 42% 4669Mi 7% 20:45:56 DEBUG --- stderr --- 20:45:56 DEBUG 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383215" 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383215" 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG { 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:55 DEBUG "data": { 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG }, 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "value": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:46:55 DEBUG 20:46:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG }, 20:46:55 DEBUG 20:46:55 DEBUG }, 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG "value": [ 20:46:55 DEBUG [...] 20:46:55 DEBUG "value": [ 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG { 20:46:55 DEBUG }, 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "data": { 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG 20:46:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "1813.7333333333333" 20:46:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:46:55 DEBUG }, 20:46:55 DEBUG "result": [ 20:46:55 DEBUG ] 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG { 20:46:55 DEBUG } 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "metric": { 20:46:55 DEBUG ] 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "1035042816" 20:46:55 DEBUG "instance": "10.142.0.107:9100" 20:46:55 DEBUG } 20:46:55 DEBUG "1814.1666666666667" 20:46:55 DEBUG ] 20:46:55 DEBUG }, 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG "value": [ 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 DEBUG 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG [...] 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG 20:46:55 DEBUG "instance": "10.142.0.43:9100" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "0.16636666666666616" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "0.01" 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "ds-idrepo-0" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "ds-cts-2" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "0" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383215" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "0" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "ds-cts-0" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "ds-cts-2" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "0" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383215" 20:46:55 ERROR [http_cmd]: ERROR 20:46:55 ERROR --- status code --- 20:46:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:46:55 ERROR --- http response --- 20:46:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:46:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "1828.1" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "instance": "10.142.0.107:9100" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "instance": "10.142.0.43:9100" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "60755450" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "6.641054703688699" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "2289.2628702726197" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "ds-idrepo-0" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "ds-idrepo-2" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "0" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "0" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "ds-cts-0" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "ds-cts-2" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "0" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "1813.3333333333333" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383215" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "328224" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383215" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "instance": "10.142.0.107:9100" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "instance": "10.142.0.43:9100" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "132193053696" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383215" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "6.686329285515461" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "instance": "10.142.0.107:9100" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "instance": "10.142.0.43:9100" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "60747741.599999994" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "ds-idrepo-0" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "ds-idrepo-2" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "0" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "ds-cts-0" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "ds-cts-2" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "29.966666666666665" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383215" 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383215" 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "instance": "10.142.0.107:9100" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "instance": "10.142.0.43:9100" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "0.08200000000000003" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "ds-idrepo-0" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "ds-idrepo-2" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "0" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:55 INFO [http_cmd]: http status code OK 20:46:55 DEBUG --- status code --- 20:46:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:46:55 DEBUG --- http response --- 20:46:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:46:55 DEBUG ----- output ----- 20:46:55 DEBUG { 20:46:55 DEBUG "status": "success", 20:46:55 DEBUG "data": { 20:46:55 DEBUG "resultType": "vector", 20:46:55 DEBUG "result": [ 20:46:55 DEBUG { 20:46:55 DEBUG "metric": { 20:46:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG [...] 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG 20:46:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:46:55 DEBUG }, 20:46:55 DEBUG "value": [ 20:46:55 DEBUG 1713383215, 20:46:55 DEBUG "0" 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG ] 20:46:55 DEBUG } 20:46:55 DEBUG } 20:46:56 INFO 20:46:56 INFO [loop_until]: kubectl --namespace=xlou top pods 20:46:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:46:56 INFO [loop_until]: OK (rc = 0) 20:46:56 DEBUG --- stdout --- 20:46:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11903m 8891Mi am-b59c765b7-bgsxp 10407m 8715Mi am-b59c765b7-ch5q4 11899m 8850Mi ds-cts-0 12382m 7695Mi ds-cts-1 11469m 7583Mi ds-cts-2 13497m 7834Mi ds-idrepo-0 5391m 3082Mi ds-idrepo-1 5075m 2758Mi ds-idrepo-2 5050m 2723Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 10m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6638m 897Mi 20:46:56 DEBUG --- stderr --- 20:46:56 DEBUG 20:46:56 INFO 20:46:56 INFO [loop_until]: kubectl --namespace=xlou top node 20:46:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:46:57 INFO [loop_until]: OK (rc = 0) 20:46:57 DEBUG --- stdout --- 20:46:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 81m 0% 5491Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5195m 17% 5971Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11765m 39% 11440Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 2981Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12484m 41% 9876Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5016Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11921m 39% 11485Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27720m 92% 2913Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11583m 38% 10560Mi 9% gke-xlou-cdm-ds-0f601751-j488 5331m 17% 5263Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11462m 38% 10996Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5626m 18% 5826Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13037m 43% 9494Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6834m 43% 4668Mi 7% 20:46:57 DEBUG --- stderr --- 20:46:57 DEBUG 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "1806.3" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383275" 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "ds-idrepo-0" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "ds-cts-2" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "0" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:55 DEBUG 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG [...] 20:47:55 DEBUG { 20:47:55 DEBUG 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG 20:47:55 DEBUG "data": { 20:47:55 DEBUG 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:47:55 DEBUG "result": [ 20:47:55 DEBUG }, 20:47:55 DEBUG { 20:47:55 DEBUG "value": [ 20:47:55 DEBUG "metric": { 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "instance": "10.142.0.107:9100" 20:47:55 DEBUG "0.01" 20:47:55 DEBUG }, 20:47:55 DEBUG ] 20:47:55 DEBUG "value": [ 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG } 20:47:55 DEBUG 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG ] 20:47:55 DEBUG 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG } 20:47:55 DEBUG 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG } 20:47:55 DEBUG [...] 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:55 DEBUG 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG 20:47:55 DEBUG { 20:47:55 DEBUG 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "instance": "10.142.0.43:9100" 20:47:55 DEBUG "data": { 20:47:55 DEBUG }, 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "value": [ 20:47:55 DEBUG "result": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG { 20:47:55 DEBUG "0.06626666666666664" 20:47:55 DEBUG "metric": { 20:47:55 DEBUG ] 20:47:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:47:55 DEBUG } 20:47:55 DEBUG }, 20:47:55 DEBUG ] 20:47:55 DEBUG "value": [ 20:47:55 DEBUG } 20:47:55 DEBUG 20:47:55 DEBUG } 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "1806.3" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "1038209024" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "instance": "10.142.0.107:9100" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "instance": "10.142.0.43:9100" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "60792566.5" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 ERROR [http_cmd]: ERROR 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 ERROR --- status code --- 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:47:55 ERROR --- http response --- 20:47:55 DEBUG "pod": "ds-cts-0" 20:47:55 DEBUG }, 20:47:55 DEBUG }, 20:47:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:47:55 DEBUG "value": [ 20:47:55 DEBUG "value": [ 20:47:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:47:55 DEBUG "pod": "ds-cts-2" 20:47:55 DEBUG }, 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "0" 20:47:55 DEBUG "0.0002992266666666666" 20:47:55 DEBUG ] 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "1820.8999999999999" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "ds-cts-0" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "12.380000070992441" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "ds-cts-0" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "286102663.0697146" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "1806.5666666666666" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "327780" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:55 DEBUG { 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG { 20:47:55 DEBUG "data": { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "result": [ 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG { 20:47:55 DEBUG "result": [ 20:47:55 DEBUG "metric": { 20:47:55 DEBUG { 20:47:55 DEBUG "pod": "ds-idrepo-0" 20:47:55 DEBUG "metric": { 20:47:55 DEBUG }, 20:47:55 DEBUG "instance": "10.142.0.107:9100" 20:47:55 DEBUG "value": [ 20:47:55 DEBUG }, 20:47:55 DEBUG 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "ds-idrepo-2" 20:47:55 DEBUG 20:47:55 DEBUG }, 20:47:55 DEBUG "instance": "10.142.0.43:9100" 20:47:55 DEBUG "value": [ 20:47:55 DEBUG }, 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG "0" 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG ] 20:47:55 DEBUG "135284937728" 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG } 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG } 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "ds-cts-0" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "35010389.89067159" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "ds-cts-0" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "ds-cts-2" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "0" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383275" 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "6.60629717440977" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "instance": "10.142.0.107:9100" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "instance": "10.142.0.43:9100" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "60784305.9" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "ds-idrepo-0" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "ds-idrepo-2" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "0" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "ds-cts-0" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "ds-cts-2" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "29.933333333333334" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383275" 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "instance": "10.142.0.107:9100" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "instance": "10.142.0.43:9100" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "0.04733333333333334" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "ds-idrepo-0" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "ds-idrepo-2" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "0" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:55 INFO [http_cmd]: http status code OK 20:47:55 DEBUG --- status code --- 20:47:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:47:55 DEBUG --- http response --- 20:47:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:47:55 DEBUG ----- output ----- 20:47:55 DEBUG { 20:47:55 DEBUG "status": "success", 20:47:55 DEBUG "data": { 20:47:55 DEBUG "resultType": "vector", 20:47:55 DEBUG "result": [ 20:47:55 DEBUG { 20:47:55 DEBUG "metric": { 20:47:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG [...] 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG 20:47:55 DEBUG "pod": "ds-cts-0" 20:47:55 DEBUG }, 20:47:55 DEBUG "value": [ 20:47:55 DEBUG 1713383275, 20:47:55 DEBUG "40786546.64205594" 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG ] 20:47:55 DEBUG } 20:47:55 DEBUG } 20:47:56 INFO 20:47:56 INFO [loop_until]: kubectl --namespace=xlou top pods 20:47:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:47:56 INFO [loop_until]: OK (rc = 0) 20:47:56 DEBUG --- stdout --- 20:47:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12159m 8893Mi am-b59c765b7-bgsxp 10989m 8713Mi am-b59c765b7-ch5q4 11873m 8740Mi ds-cts-0 12237m 7710Mi ds-cts-1 11972m 7563Mi ds-cts-2 13201m 7840Mi ds-idrepo-0 5181m 3082Mi ds-idrepo-1 5209m 2758Mi ds-idrepo-2 5066m 2723Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 10m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6699m 898Mi 20:47:56 DEBUG --- stderr --- 20:47:56 DEBUG 20:47:57 INFO 20:47:57 INFO [loop_until]: kubectl --namespace=xlou top node 20:47:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:47:57 INFO [loop_until]: OK (rc = 0) 20:47:57 DEBUG --- stdout --- 20:47:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 5488Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5136m 17% 5958Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11726m 39% 11326Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 77m 0% 2982Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12340m 41% 9891Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5022Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12013m 40% 11494Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28026m 93% 2915Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11613m 38% 10552Mi 9% gke-xlou-cdm-ds-0f601751-j488 5405m 18% 5267Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11710m 39% 11004Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5635m 18% 5823Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12923m 43% 9502Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6778m 42% 4666Mi 7% 20:47:57 DEBUG --- stderr --- 20:47:57 DEBUG 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "ds-idrepo-0" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "ds-cts-2" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "0" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "0.01" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383335" 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "1812.6333333333334" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "1812.6333333333332" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "instance": "10.142.0.107:9100" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "instance": "10.142.0.43:9100" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "0.161166666666666" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "1041760256" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383335" 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "ds-cts-0" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "ds-cts-2" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "0" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "instance": "10.142.0.107:9100" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "instance": "10.142.0.43:9100" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "60825081.2" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "1830.5666666666666" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383335" 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "0" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 ERROR [http_cmd]: ERROR 20:48:55 ERROR --- status code --- 20:48:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:48:55 ERROR --- http response --- 20:48:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:48:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "ds-idrepo-1" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "idm-b85f9d978-txpwv" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "0.007101079326322979" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "ds-idrepo-1" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "idm-b85f9d978-txpwv" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "1704.7251741072669" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "ds-idrepo-1" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "idm-b85f9d978-txpwv" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "0" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "1812.3666666666666" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "ds-idrepo-0" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "ds-idrepo-2" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "0" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "327889" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383335" 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "instance": "10.142.0.107:9100" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "instance": "10.142.0.43:9100" 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:48:55 DEBUG }, 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG "value": [ 20:48:55 DEBUG { 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "138045795328" 20:48:55 DEBUG "data": { 20:48:55 DEBUG ] 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG } 20:48:55 DEBUG "result": [ 20:48:55 DEBUG ] 20:48:55 DEBUG { 20:48:55 DEBUG } 20:48:55 DEBUG "metric": { 20:48:55 DEBUG } 20:48:55 DEBUG "pod": "ds-cts-0" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "ds-cts-2" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "0" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383335" 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "6.615173987186581" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "instance": "10.142.0.107:9100" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "instance": "10.142.0.43:9100" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "60817789.2" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "ds-idrepo-0" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "ds-idrepo-2" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "0" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "ds-cts-0" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "ds-cts-2" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "29.966666666666665" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG "result": [ 20:48:55 DEBUG { 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "pod": "ds-idrepo-0" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "ds-idrepo-2" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "0" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383335" 20:48:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383335" 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG { 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG "data": { 20:48:55 INFO [http_cmd]: http status code OK 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG --- status code --- 20:48:55 DEBUG "result": [ 20:48:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:48:55 DEBUG { 20:48:55 DEBUG --- http response --- 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "instance": "10.142.0.107:9100" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:48:55 DEBUG [...] 20:48:55 DEBUG ----- output ----- 20:48:55 DEBUG 20:48:55 DEBUG { 20:48:55 DEBUG 20:48:55 DEBUG "status": "success", 20:48:55 DEBUG 20:48:55 DEBUG "data": { 20:48:55 DEBUG "instance": "10.142.0.43:9100" 20:48:55 DEBUG "resultType": "vector", 20:48:55 DEBUG }, 20:48:55 DEBUG "result": [ 20:48:55 DEBUG "value": [ 20:48:55 DEBUG { 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "metric": { 20:48:55 DEBUG "0.0766666666666668" 20:48:55 DEBUG "pod": "ds-idrepo-1" 20:48:55 DEBUG ] 20:48:55 DEBUG }, 20:48:55 DEBUG } 20:48:55 DEBUG "value": [ 20:48:55 DEBUG ] 20:48:55 DEBUG 20:48:55 DEBUG } 20:48:55 DEBUG 20:48:55 DEBUG } 20:48:55 DEBUG 20:48:55 DEBUG [...] 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG 20:48:55 DEBUG "pod": "idm-b85f9d978-txpwv" 20:48:55 DEBUG }, 20:48:55 DEBUG "value": [ 20:48:55 DEBUG 1713383335, 20:48:55 DEBUG "909.1867595238757" 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG ] 20:48:55 DEBUG } 20:48:55 DEBUG } 20:48:56 INFO 20:48:56 INFO [loop_until]: kubectl --namespace=xlou top pods 20:48:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:48:56 INFO [loop_until]: OK (rc = 0) 20:48:56 DEBUG --- stdout --- 20:48:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11939m 8893Mi am-b59c765b7-bgsxp 10861m 8713Mi am-b59c765b7-ch5q4 12033m 8740Mi ds-cts-0 12743m 7695Mi ds-cts-1 11364m 7589Mi ds-cts-2 12954m 7865Mi ds-idrepo-0 5266m 3082Mi ds-idrepo-1 5214m 2758Mi ds-idrepo-2 5024m 2723Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6516m 900Mi 20:48:56 DEBUG --- stderr --- 20:48:56 DEBUG 20:48:57 INFO 20:48:57 INFO [loop_until]: kubectl --namespace=xlou top node 20:48:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:48:57 INFO [loop_until]: OK (rc = 0) 20:48:57 DEBUG --- stdout --- 20:48:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 5498Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4979m 16% 5966Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11726m 39% 11324Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 2997Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12501m 41% 9889Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5029Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11607m 38% 11494Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27155m 90% 2914Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11590m 38% 10548Mi 9% gke-xlou-cdm-ds-0f601751-j488 5338m 17% 5259Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11575m 38% 11001Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5601m 18% 5811Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12902m 43% 9501Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6776m 42% 4666Mi 7% 20:48:57 DEBUG --- stderr --- 20:48:57 DEBUG 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "1826.7" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "ds-idrepo-0" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "ds-cts-2" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "0" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "0.01" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383395" 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "1043910656" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "1826.9" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "instance": "10.142.0.107:9100" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "instance": "10.142.0.43:9100" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "0.05216666666666564" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "1813.7666666666667" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "instance": "10.142.0.107:9100" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "instance": "10.142.0.43:9100" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "61054055.63333333" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG } 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "ds-cts-0" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "ds-cts-2" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "0.00006666666666666667" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "0" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 ERROR [http_cmd]: ERROR 20:49:55 DEBUG --- status code --- 20:49:55 ERROR --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 ERROR --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "0" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:49:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "0.0000231788684719517" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "instance": "10.142.0.107:9100" 20:49:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:49:55 DEBUG }, 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "instance": "10.142.0.43:9100" 20:49:55 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:49:55 DEBUG }, 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "140606323712" 20:49:55 DEBUG "0" 20:49:55 DEBUG ] 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "1826.5333333333333" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "ds-idrepo-0" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "ds-idrepo-2" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "0" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "327676" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "ds-cts-0" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "ds-cts-2" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "0" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383395" 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "ds-cts-0" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "ds-cts-2" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "29.966666666666665" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "instance": "10.142.0.107:9100" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "instance": "10.142.0.43:9100" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "61045979.266666666" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "6.660856902313951" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "ds-idrepo-0" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "ds-idrepo-2" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "0" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: http status code OK 20:49:55 DEBUG --- status code --- 20:49:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:55 DEBUG --- http response --- 20:49:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:49:55 DEBUG ----- output ----- 20:49:55 DEBUG { 20:49:55 DEBUG "status": "success", 20:49:55 DEBUG "data": { 20:49:55 DEBUG "resultType": "vector", 20:49:55 DEBUG "result": [ 20:49:55 DEBUG { 20:49:55 DEBUG "metric": { 20:49:55 DEBUG "pod": "ds-idrepo-0" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG [...] 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG 20:49:55 DEBUG "pod": "ds-idrepo-2" 20:49:55 DEBUG }, 20:49:55 DEBUG "value": [ 20:49:55 DEBUG 1713383395, 20:49:55 DEBUG "0" 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG ] 20:49:55 DEBUG } 20:49:55 DEBUG } 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383395" 20:49:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383395" 20:49:56 INFO [http_cmd]: http status code OK 20:49:56 DEBUG --- status code --- 20:49:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:56 DEBUG --- http response --- 20:49:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:56 DEBUG ----- output ----- 20:49:56 DEBUG { 20:49:56 DEBUG "status": "success", 20:49:56 DEBUG "data": { 20:49:56 DEBUG "resultType": "vector", 20:49:56 DEBUG "result": [ 20:49:56 DEBUG { 20:49:56 DEBUG "metric": { 20:49:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:49:56 DEBUG }, 20:49:56 DEBUG "value": [ 20:49:56 DEBUG 20:49:56 DEBUG 20:49:56 DEBUG 20:49:56 DEBUG [...] 20:49:56 DEBUG 20:49:56 DEBUG 20:49:56 DEBUG 20:49:56 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:49:56 DEBUG }, 20:49:56 DEBUG "value": [ 20:49:56 DEBUG 1713383395, 20:49:56 DEBUG "0" 20:49:56 DEBUG ] 20:49:56 DEBUG } 20:49:56 DEBUG ] 20:49:56 DEBUG } 20:49:56 DEBUG } 20:49:56 INFO [http_cmd]: http status code OK 20:49:56 DEBUG --- status code --- 20:49:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:49:56 DEBUG --- http response --- 20:49:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:49:56 DEBUG ----- output ----- 20:49:56 DEBUG { 20:49:56 DEBUG "status": "success", 20:49:56 DEBUG "data": { 20:49:56 DEBUG "resultType": "vector", 20:49:56 DEBUG "result": [ 20:49:56 DEBUG { 20:49:56 DEBUG "metric": { 20:49:56 DEBUG "instance": "10.142.0.107:9100" 20:49:56 DEBUG }, 20:49:56 DEBUG "value": [ 20:49:56 DEBUG 20:49:56 DEBUG 20:49:56 DEBUG 20:49:56 DEBUG [...] 20:49:56 DEBUG 20:49:56 DEBUG 20:49:56 DEBUG 20:49:56 DEBUG "instance": "10.142.0.43:9100" 20:49:56 DEBUG }, 20:49:56 DEBUG "value": [ 20:49:56 DEBUG 1713383395, 20:49:56 DEBUG "0.03500000000000013" 20:49:56 DEBUG ] 20:49:56 DEBUG } 20:49:56 DEBUG ] 20:49:56 DEBUG } 20:49:56 DEBUG } 20:49:56 INFO 20:49:56 INFO [loop_until]: kubectl --namespace=xlou top pods 20:49:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:49:56 INFO [loop_until]: OK (rc = 0) 20:49:56 DEBUG --- stdout --- 20:49:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11447m 8892Mi am-b59c765b7-bgsxp 11058m 8711Mi am-b59c765b7-ch5q4 11831m 8741Mi ds-cts-0 12701m 7702Mi ds-cts-1 11625m 7581Mi ds-cts-2 12712m 7845Mi ds-idrepo-0 5320m 3081Mi ds-idrepo-1 5204m 2758Mi ds-idrepo-2 5087m 2723Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 10m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6469m 906Mi 20:49:56 DEBUG --- stderr --- 20:49:56 DEBUG 20:49:57 INFO 20:49:57 INFO [loop_until]: kubectl --namespace=xlou top node 20:49:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:49:57 INFO [loop_until]: OK (rc = 0) 20:49:57 DEBUG --- stdout --- 20:49:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 5492Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5133m 17% 5964Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11752m 39% 11319Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 2988Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12525m 41% 9901Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5030Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11968m 40% 11491Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28023m 93% 2904Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11675m 39% 10552Mi 9% gke-xlou-cdm-ds-0f601751-j488 5393m 18% 5258Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11782m 39% 11019Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5599m 18% 5822Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12792m 42% 9492Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6789m 42% 4666Mi 7% 20:49:57 DEBUG --- stderr --- 20:49:57 DEBUG 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "1828.6666666666667" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "1828.9333333333334" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383455" 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "instance": "10.142.0.107:9100" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "instance": "10.142.0.43:9100" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "0.11283333333333302" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "ds-idrepo-0" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "ds-cts-2" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "0" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383455" 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG { 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "0.01" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "1806.6333333333332" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "1045192704" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383455" 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383455" 20:50:55 ERROR [http_cmd]: ERROR 20:50:55 ERROR --- status code --- 20:50:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:50:55 ERROR --- http response --- 20:50:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:50:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "instance": "10.142.0.107:9100" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "instance": "10.142.0.43:9100" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "60822781.233333334" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "ds-cts-0" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "ds-cts-2" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "0.00023333333333333333" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "51636.76462938881" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "0" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "3.557001235649811" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383455" 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "instance": "10.142.0.107:9100" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "instance": "10.142.0.43:9100" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "143395066880" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "0" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "327858" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "1829.2" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "ds-cts-0" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "ds-cts-2" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "0" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "ds-idrepo-0" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "pod": "ds-idrepo-2" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "0" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383455" 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383455" 20:50:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383455" 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "instance": "10.142.0.107:9100" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "instance": "10.142.0.43:9100" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "60814809.233333334" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "pod": "ds-cts-0" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 INFO [http_cmd]: http status code OK 20:50:55 DEBUG 20:50:55 DEBUG --- status code --- 20:50:55 DEBUG 20:50:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:55 DEBUG "pod": "ds-cts-2" 20:50:55 DEBUG --- http response --- 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "29.9" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:55 DEBUG ----- output ----- 20:50:55 DEBUG { 20:50:55 DEBUG "status": "success", 20:50:55 DEBUG "data": { 20:50:55 DEBUG "resultType": "vector", 20:50:55 DEBUG "result": [ 20:50:55 DEBUG { 20:50:55 DEBUG "metric": { 20:50:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG [...] 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG 20:50:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:50:55 DEBUG }, 20:50:55 DEBUG "value": [ 20:50:55 DEBUG 1713383455, 20:50:55 DEBUG "6.610178346930325" 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG ] 20:50:55 DEBUG } 20:50:55 DEBUG } 20:50:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383456" 20:50:56 INFO [http_cmd]: http status code OK 20:50:56 DEBUG --- status code --- 20:50:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:56 DEBUG --- http response --- 20:50:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:56 DEBUG ----- output ----- 20:50:56 DEBUG { 20:50:56 DEBUG "status": "success", 20:50:56 DEBUG "data": { 20:50:56 DEBUG "resultType": "vector", 20:50:56 DEBUG "result": [ 20:50:56 DEBUG { 20:50:56 DEBUG "metric": { 20:50:56 DEBUG "pod": "ds-idrepo-0" 20:50:56 DEBUG }, 20:50:56 DEBUG "value": [ 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG [...] 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG "pod": "ds-idrepo-2" 20:50:56 DEBUG }, 20:50:56 DEBUG "value": [ 20:50:56 DEBUG 1713383456, 20:50:56 DEBUG "0" 20:50:56 DEBUG ] 20:50:56 DEBUG } 20:50:56 DEBUG ] 20:50:56 DEBUG } 20:50:56 DEBUG } 20:50:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383456" 20:50:56 INFO [http_cmd]: http status code OK 20:50:56 DEBUG --- status code --- 20:50:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:56 DEBUG --- http response --- 20:50:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:50:56 DEBUG ----- output ----- 20:50:56 DEBUG { 20:50:56 DEBUG "status": "success", 20:50:56 DEBUG "data": { 20:50:56 DEBUG "resultType": "vector", 20:50:56 DEBUG "result": [ 20:50:56 DEBUG { 20:50:56 DEBUG "metric": { 20:50:56 DEBUG "pod": "ds-idrepo-0" 20:50:56 DEBUG }, 20:50:56 DEBUG "value": [ 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG [...] 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG "pod": "ds-idrepo-2" 20:50:56 DEBUG }, 20:50:56 DEBUG "value": [ 20:50:56 DEBUG 1713383456, 20:50:56 DEBUG "0" 20:50:56 DEBUG ] 20:50:56 DEBUG } 20:50:56 DEBUG ] 20:50:56 DEBUG } 20:50:56 DEBUG } 20:50:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383456" 20:50:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383456" 20:50:56 INFO [http_cmd]: http status code OK 20:50:56 DEBUG --- status code --- 20:50:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:56 DEBUG --- http response --- 20:50:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:56 DEBUG ----- output ----- 20:50:56 DEBUG { 20:50:56 DEBUG "status": "success", 20:50:56 DEBUG "data": { 20:50:56 DEBUG "resultType": "vector", 20:50:56 DEBUG "result": [ 20:50:56 DEBUG { 20:50:56 DEBUG "metric": { 20:50:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:50:56 DEBUG }, 20:50:56 DEBUG "value": [ 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG [...] 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:50:56 DEBUG }, 20:50:56 DEBUG "value": [ 20:50:56 DEBUG 1713383456, 20:50:56 DEBUG "0" 20:50:56 DEBUG ] 20:50:56 DEBUG } 20:50:56 DEBUG ] 20:50:56 DEBUG } 20:50:56 DEBUG } 20:50:56 INFO [http_cmd]: http status code OK 20:50:56 DEBUG --- status code --- 20:50:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:50:56 DEBUG --- http response --- 20:50:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:50:56 DEBUG ----- output ----- 20:50:56 DEBUG { 20:50:56 DEBUG "status": "success", 20:50:56 DEBUG "data": { 20:50:56 DEBUG "resultType": "vector", 20:50:56 DEBUG "result": [ 20:50:56 DEBUG { 20:50:56 DEBUG "metric": { 20:50:56 DEBUG "instance": "10.142.0.107:9100" 20:50:56 DEBUG }, 20:50:56 DEBUG "value": [ 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG [...] 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG 20:50:56 DEBUG "instance": "10.142.0.43:9100" 20:50:56 DEBUG }, 20:50:56 DEBUG "value": [ 20:50:56 DEBUG 1713383456, 20:50:56 DEBUG "0.057999999999999885" 20:50:56 DEBUG ] 20:50:56 DEBUG } 20:50:56 DEBUG ] 20:50:56 DEBUG } 20:50:56 DEBUG } 20:50:56 INFO 20:50:56 INFO [loop_until]: kubectl --namespace=xlou top pods 20:50:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:50:56 INFO [loop_until]: OK (rc = 0) 20:50:56 DEBUG --- stdout --- 20:50:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11900m 8890Mi am-b59c765b7-bgsxp 11120m 8715Mi am-b59c765b7-ch5q4 11904m 8744Mi ds-cts-0 12519m 7699Mi ds-cts-1 11617m 7577Mi ds-cts-2 12339m 7871Mi ds-idrepo-0 5255m 3082Mi ds-idrepo-1 5085m 2758Mi ds-idrepo-2 4958m 2723Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 10m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6436m 904Mi 20:50:56 DEBUG --- stderr --- 20:50:56 DEBUG 20:50:57 INFO 20:50:57 INFO [loop_until]: kubectl --namespace=xlou top node 20:50:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:50:57 INFO [loop_until]: OK (rc = 0) 20:50:57 DEBUG --- stdout --- 20:50:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 5490Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5141m 17% 5962Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11264m 37% 11320Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 2991Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12466m 41% 9905Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5008Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11949m 40% 11498Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27918m 93% 2928Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11708m 39% 10542Mi 9% gke-xlou-cdm-ds-0f601751-j488 5417m 18% 5252Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11681m 39% 11013Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 5634m 18% 5840Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13130m 43% 9503Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6764m 42% 4671Mi 7% 20:50:57 DEBUG --- stderr --- 20:50:57 DEBUG 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "1824.0666666666666" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383515" 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "1824.2666666666667" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "instance": "10.142.0.107:9100" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "instance": "10.142.0.43:9100" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "0.08530000000000086" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "1812.5333333333333" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383515" 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "ds-idrepo-0" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "ds-cts-2" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "0" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "0.011" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383515" 20:51:55 ERROR [http_cmd]: ERROR 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 ERROR --- status code --- 20:51:55 DEBUG --- status code --- 20:51:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 ERROR --- http response --- 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:51:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:51:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "ds-idrepo-1" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "81261.05177774774" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "1047838720" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "ds-cts-0" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "ds-cts-2" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "0.00008730666666666666" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383515" 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "ds-idrepo-1" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "6.68389536041631" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "instance": "10.142.0.107:9100" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "instance": "10.142.0.43:9100" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "60809487.03333333" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "0" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383515" 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "instance": "10.142.0.107:9100" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "instance": "10.142.0.43:9100" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "146388575232" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "1823.8666666666666" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "ds-idrepo-1" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "0" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "ds-cts-0" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "ds-cts-2" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "0" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383515" 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "328523" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:55 INFO [http_cmd]: http status code OK 20:51:55 DEBUG --- status code --- 20:51:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:55 DEBUG --- http response --- 20:51:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:55 DEBUG ----- output ----- 20:51:55 DEBUG { 20:51:55 DEBUG "status": "success", 20:51:55 DEBUG "data": { 20:51:55 DEBUG "resultType": "vector", 20:51:55 DEBUG "result": [ 20:51:55 DEBUG { 20:51:55 DEBUG "metric": { 20:51:55 DEBUG "pod": "ds-idrepo-0" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG [...] 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG 20:51:55 DEBUG "pod": "ds-idrepo-2" 20:51:55 DEBUG }, 20:51:55 DEBUG "value": [ 20:51:55 DEBUG 1713383515, 20:51:55 DEBUG "0" 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG ] 20:51:55 DEBUG } 20:51:55 DEBUG } 20:51:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383516" 20:51:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383516" 20:51:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383516" 20:51:56 INFO [http_cmd]: http status code OK 20:51:56 DEBUG --- status code --- 20:51:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:56 DEBUG --- http response --- 20:51:56 INFO [http_cmd]: http status code OK 20:51:56 INFO [http_cmd]: http status code OK 20:51:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:51:56 DEBUG --- status code --- 20:51:56 DEBUG --- status code --- 20:51:56 DEBUG ----- output ----- 20:51:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:56 DEBUG { 20:51:56 DEBUG --- http response --- 20:51:56 DEBUG --- http response --- 20:51:56 DEBUG "status": "success", 20:51:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:56 DEBUG "data": { 20:51:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:56 DEBUG ----- output ----- 20:51:56 DEBUG "resultType": "vector", 20:51:56 DEBUG ----- output ----- 20:51:56 DEBUG { 20:51:56 DEBUG "result": [ 20:51:56 DEBUG { 20:51:56 DEBUG "status": "success", 20:51:56 DEBUG { 20:51:56 DEBUG "status": "success", 20:51:56 DEBUG "data": { 20:51:56 DEBUG "metric": { 20:51:56 DEBUG "data": { 20:51:56 DEBUG "resultType": "vector", 20:51:56 DEBUG "pod": "ds-cts-0" 20:51:56 DEBUG "resultType": "vector", 20:51:56 DEBUG "result": [ 20:51:56 DEBUG }, 20:51:56 DEBUG "result": [ 20:51:56 DEBUG { 20:51:56 DEBUG "value": [ 20:51:56 DEBUG { 20:51:56 DEBUG "metric": { 20:51:56 DEBUG 20:51:56 DEBUG "metric": { 20:51:56 DEBUG "instance": "10.142.0.107:9100" 20:51:56 DEBUG 20:51:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:51:56 DEBUG }, 20:51:56 DEBUG 20:51:56 DEBUG }, 20:51:56 DEBUG "value": [ 20:51:56 DEBUG [...] 20:51:56 DEBUG "value": [ 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG [...] 20:51:56 DEBUG "pod": "ds-cts-2" 20:51:56 DEBUG [...] 20:51:56 DEBUG 20:51:56 DEBUG }, 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG "value": [ 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 1713383516, 20:51:56 DEBUG 20:51:56 DEBUG "instance": "10.142.0.43:9100" 20:51:56 DEBUG "30" 20:51:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:51:56 DEBUG }, 20:51:56 DEBUG ] 20:51:56 DEBUG }, 20:51:56 DEBUG "value": [ 20:51:56 DEBUG } 20:51:56 DEBUG "value": [ 20:51:56 DEBUG 1713383516, 20:51:56 DEBUG ] 20:51:56 DEBUG 1713383516, 20:51:56 DEBUG "60803561.13333333" 20:51:56 DEBUG } 20:51:56 DEBUG "6.70662918191221" 20:51:56 DEBUG ] 20:51:56 DEBUG } 20:51:56 DEBUG ] 20:51:56 DEBUG } 20:51:56 DEBUG } 20:51:56 DEBUG ] 20:51:56 DEBUG ] 20:51:56 DEBUG } 20:51:56 DEBUG } 20:51:56 DEBUG } 20:51:56 DEBUG } 20:51:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383516" 20:51:56 INFO [http_cmd]: http status code OK 20:51:56 DEBUG --- status code --- 20:51:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:56 DEBUG --- http response --- 20:51:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:56 DEBUG ----- output ----- 20:51:56 DEBUG { 20:51:56 DEBUG "status": "success", 20:51:56 DEBUG "data": { 20:51:56 DEBUG "resultType": "vector", 20:51:56 DEBUG "result": [ 20:51:56 DEBUG { 20:51:56 DEBUG "metric": { 20:51:56 DEBUG "pod": "ds-idrepo-0" 20:51:56 DEBUG }, 20:51:56 DEBUG "value": [ 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG [...] 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG "pod": "ds-idrepo-2" 20:51:56 DEBUG }, 20:51:56 DEBUG "value": [ 20:51:56 DEBUG 1713383516, 20:51:56 DEBUG "0" 20:51:56 DEBUG ] 20:51:56 DEBUG } 20:51:56 DEBUG ] 20:51:56 DEBUG } 20:51:56 DEBUG } 20:51:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383516" 20:51:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383516" 20:51:56 INFO [http_cmd]: http status code OK 20:51:56 DEBUG --- status code --- 20:51:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:56 DEBUG --- http response --- 20:51:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:56 DEBUG ----- output ----- 20:51:56 DEBUG { 20:51:56 DEBUG "status": "success", 20:51:56 DEBUG "data": { 20:51:56 DEBUG "resultType": "vector", 20:51:56 DEBUG "result": [ 20:51:56 DEBUG { 20:51:56 DEBUG "metric": { 20:51:56 DEBUG "pod": "ds-idrepo-1" 20:51:56 DEBUG }, 20:51:56 DEBUG "value": [ 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG [...] 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:51:56 DEBUG }, 20:51:56 DEBUG "value": [ 20:51:56 DEBUG 1713383516, 20:51:56 DEBUG "0" 20:51:56 DEBUG ] 20:51:56 DEBUG } 20:51:56 DEBUG ] 20:51:56 DEBUG } 20:51:56 DEBUG } 20:51:56 INFO [http_cmd]: http status code OK 20:51:56 DEBUG --- status code --- 20:51:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:56 DEBUG --- http response --- 20:51:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:51:56 DEBUG ----- output ----- 20:51:56 DEBUG { 20:51:56 DEBUG "status": "success", 20:51:56 DEBUG "data": { 20:51:56 DEBUG "resultType": "vector", 20:51:56 DEBUG "result": [ 20:51:56 DEBUG { 20:51:56 DEBUG "metric": { 20:51:56 DEBUG "pod": "ds-idrepo-0" 20:51:56 DEBUG }, 20:51:56 DEBUG "value": [ 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG [...] 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG "pod": "ds-idrepo-2" 20:51:56 DEBUG }, 20:51:56 DEBUG "value": [ 20:51:56 DEBUG 1713383516, 20:51:56 DEBUG "0" 20:51:56 DEBUG ] 20:51:56 DEBUG } 20:51:56 DEBUG ] 20:51:56 DEBUG } 20:51:56 DEBUG } 20:51:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383516" 20:51:56 INFO [http_cmd]: http status code OK 20:51:56 DEBUG --- status code --- 20:51:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:51:56 DEBUG --- http response --- 20:51:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:51:56 DEBUG ----- output ----- 20:51:56 DEBUG { 20:51:56 DEBUG "status": "success", 20:51:56 DEBUG "data": { 20:51:56 DEBUG "resultType": "vector", 20:51:56 DEBUG "result": [ 20:51:56 DEBUG { 20:51:56 DEBUG "metric": { 20:51:56 DEBUG "instance": "10.142.0.107:9100" 20:51:56 DEBUG }, 20:51:56 DEBUG "value": [ 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG [...] 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG 20:51:56 DEBUG "instance": "10.142.0.43:9100" 20:51:56 DEBUG }, 20:51:56 DEBUG "value": [ 20:51:56 DEBUG 1713383516, 20:51:56 DEBUG "0.05633333333333332" 20:51:56 DEBUG ] 20:51:56 DEBUG } 20:51:56 DEBUG ] 20:51:56 DEBUG } 20:51:56 DEBUG } 20:51:56 INFO 20:51:56 INFO [loop_until]: kubectl --namespace=xlou top pods 20:51:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:51:56 INFO [loop_until]: OK (rc = 0) 20:51:56 DEBUG --- stdout --- 20:51:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11810m 8891Mi am-b59c765b7-bgsxp 10830m 8711Mi am-b59c765b7-ch5q4 11959m 8743Mi ds-cts-0 11967m 7725Mi ds-cts-1 11716m 7602Mi ds-cts-2 13549m 7864Mi ds-idrepo-0 5193m 3082Mi ds-idrepo-1 5237m 2758Mi ds-idrepo-2 5138m 2723Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 966Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 7m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6732m 907Mi 20:51:56 DEBUG --- stderr --- 20:51:56 DEBUG 20:51:57 INFO 20:51:57 INFO [loop_until]: kubectl --namespace=xlou top node 20:51:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:51:57 INFO [loop_until]: OK (rc = 0) 20:51:57 DEBUG --- stdout --- 20:51:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 5493Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3621m 12% 5955Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9401m 31% 11332Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 2970Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12013m 40% 9916Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5016Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9654m 32% 11509Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 20972m 70% 2906Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10382m 34% 10553Mi 9% gke-xlou-cdm-ds-0f601751-j488 5226m 17% 5263Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 9604m 32% 11021Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 4577m 15% 5827Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 10567m 35% 9518Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 5704m 35% 4113Mi 7% 20:51:57 DEBUG --- stderr --- 20:51:57 DEBUG 20:52:07 WARNING File /tmp/lodemon_data-20240417_201944/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [17/Apr/2024 20:52:07] "GET /monitoring/average?start_time=20240417_192256&stop_time=20240417_195007 HTTP/1.1" 200 - 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383575" 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383575" 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383575" 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383575" 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 DEBUG --- status code --- 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG --- http response --- 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG --- http response --- 20:52:55 DEBUG --- http response --- 20:52:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:55 DEBUG --- http response --- 20:52:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:55 DEBUG ----- output ----- 20:52:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:55 DEBUG ----- output ----- 20:52:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:55 DEBUG { 20:52:55 DEBUG ----- output ----- 20:52:55 DEBUG { 20:52:55 DEBUG ----- output ----- 20:52:55 DEBUG "status": "success", 20:52:55 DEBUG { 20:52:55 DEBUG "status": "success", 20:52:55 DEBUG { 20:52:55 DEBUG "data": { 20:52:55 DEBUG "status": "success", 20:52:55 DEBUG "data": { 20:52:55 DEBUG "status": "success", 20:52:55 DEBUG "resultType": "vector", 20:52:55 DEBUG "data": { 20:52:55 DEBUG "resultType": "vector", 20:52:55 DEBUG "data": { 20:52:55 DEBUG "result": [ 20:52:55 DEBUG "resultType": "vector", 20:52:55 DEBUG "result": [ 20:52:55 DEBUG "resultType": "vector", 20:52:55 DEBUG { 20:52:55 DEBUG "result": [ 20:52:55 DEBUG { 20:52:55 DEBUG "result": [ 20:52:55 DEBUG "metric": { 20:52:55 DEBUG { 20:52:55 DEBUG "metric": { 20:52:55 DEBUG { 20:52:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:52:55 DEBUG "metric": { 20:52:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:52:55 DEBUG "metric": { 20:52:55 DEBUG }, 20:52:55 DEBUG "instance": "10.142.0.107:9100" 20:52:55 DEBUG }, 20:52:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:52:55 DEBUG "value": [ 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG }, 20:52:55 DEBUG 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG [...] 20:52:55 DEBUG 20:52:55 DEBUG [...] 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG [...] 20:52:55 DEBUG 20:52:55 DEBUG [...] 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:52:55 DEBUG 20:52:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:52:55 DEBUG 20:52:55 DEBUG }, 20:52:55 DEBUG "instance": "10.142.0.43:9100" 20:52:55 DEBUG }, 20:52:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:52:55 DEBUG "value": [ 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG }, 20:52:55 DEBUG 1713383575, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 1713383575, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG "1825.9333333333334" 20:52:55 DEBUG 1713383575, 20:52:55 DEBUG "306.6333333333333" 20:52:55 DEBUG 1713383575, 20:52:55 DEBUG ] 20:52:55 DEBUG "0.12283333333333435" 20:52:55 DEBUG ] 20:52:55 DEBUG "306.5333333333333" 20:52:55 DEBUG } 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG ] 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383575" 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383575" 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383575" 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG --- http response --- 20:52:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:52:55 DEBUG ----- output ----- 20:52:55 DEBUG { 20:52:55 DEBUG "status": "success", 20:52:55 DEBUG "data": { 20:52:55 DEBUG "resultType": "vector", 20:52:55 DEBUG "result": [ 20:52:55 DEBUG { 20:52:55 DEBUG "metric": { 20:52:55 DEBUG "pod": "ds-idrepo-1" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG [...] 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 1713383575, 20:52:55 DEBUG "11845219.015686743" 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383575" 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG --- http response --- 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383575" 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383575" 20:52:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:52:55 DEBUG ----- output ----- 20:52:55 DEBUG { 20:52:55 DEBUG "status": "success", 20:52:55 DEBUG "data": { 20:52:55 DEBUG "resultType": "vector", 20:52:55 DEBUG "result": [ 20:52:55 DEBUG { 20:52:55 DEBUG "metric": { 20:52:55 DEBUG "pod": "ds-cts-0" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG [...] 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG "pod": "ds-cts-2" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 1713383575, 20:52:55 DEBUG "0.000033333333333333335" 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG --- http response --- 20:52:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:55 DEBUG ----- output ----- 20:52:55 DEBUG { 20:52:55 DEBUG "status": "success", 20:52:55 DEBUG "data": { 20:52:55 DEBUG "resultType": "vector", 20:52:55 DEBUG "result": [ 20:52:55 DEBUG { 20:52:55 DEBUG "metric": { 20:52:55 DEBUG "instance": "10.142.0.107:9100" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG [...] 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG "instance": "10.142.0.43:9100" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 1713383575, 20:52:55 DEBUG "16694921.266666666" 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383575" 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG --- http response --- 20:52:55 DEBUG --- http response --- 20:52:55 DEBUG --- http response --- 20:52:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:52:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:52:55 DEBUG ----- output ----- 20:52:55 DEBUG ----- output ----- 20:52:55 DEBUG { 20:52:55 DEBUG { 20:52:55 DEBUG "status": "success", 20:52:55 DEBUG "status": "success", 20:52:55 DEBUG "data": { 20:52:55 DEBUG "data": { 20:52:55 DEBUG "resultType": "vector", 20:52:55 DEBUG "resultType": "vector", 20:52:55 DEBUG "result": [ 20:52:55 DEBUG "result": [ 20:52:55 DEBUG { 20:52:55 DEBUG { 20:52:55 DEBUG "metric": { 20:52:55 DEBUG "metric": { 20:52:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:52:55 DEBUG "pod": "ds-idrepo-0" 20:52:55 DEBUG }, 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG [...] 20:52:55 DEBUG [...] 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:52:55 DEBUG "pod": "ds-cts-2" 20:52:55 DEBUG }, 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 1713383575, 20:52:55 DEBUG 1713383575, 20:52:55 DEBUG "0.012" 20:52:55 DEBUG "0" 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383575" 20:52:55 DEBUG ] 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 DEBUG ] 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 ERROR [http_cmd]: ERROR 20:52:55 ERROR --- status code --- 20:52:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:52:55 ERROR --- http response --- 20:52:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:52:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG --- http response --- 20:52:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:55 DEBUG ----- output ----- 20:52:55 DEBUG { 20:52:55 DEBUG "status": "success", 20:52:55 DEBUG "data": { 20:52:55 DEBUG "resultType": "vector", 20:52:55 DEBUG "result": [ 20:52:55 DEBUG { 20:52:55 DEBUG "metric": { 20:52:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG [...] 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 1713383575, 20:52:55 DEBUG "578949120" 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383575" 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG --- http response --- 20:52:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:52:55 DEBUG ----- output ----- 20:52:55 DEBUG { 20:52:55 DEBUG "status": "success", 20:52:55 DEBUG "data": { 20:52:55 DEBUG "resultType": "vector", 20:52:55 DEBUG "result": [ 20:52:55 DEBUG { 20:52:55 DEBUG "metric": { 20:52:55 DEBUG "pod": "ds-idrepo-1" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG [...] 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 1713383575, 20:52:55 DEBUG "1.7676169517123461" 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383575" 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG --- http response --- 20:52:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:55 DEBUG ----- output ----- 20:52:55 DEBUG { 20:52:55 DEBUG "status": "success", 20:52:55 DEBUG "data": { 20:52:55 DEBUG "resultType": "vector", 20:52:55 DEBUG "result": [ 20:52:55 DEBUG { 20:52:55 DEBUG "metric": { 20:52:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG [...] 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 1713383575, 20:52:55 DEBUG "0" 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383575" 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG --- http response --- 20:52:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:55 DEBUG ----- output ----- 20:52:55 DEBUG { 20:52:55 DEBUG "status": "success", 20:52:55 DEBUG "data": { 20:52:55 DEBUG "resultType": "vector", 20:52:55 DEBUG "result": [ 20:52:55 DEBUG { 20:52:55 DEBUG "metric": { 20:52:55 DEBUG "instance": "10.142.0.107:9100" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG [...] 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG 20:52:55 DEBUG "instance": "10.142.0.43:9100" 20:52:55 DEBUG }, 20:52:55 DEBUG "value": [ 20:52:55 DEBUG 1713383575, 20:52:55 DEBUG "147874683904" 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG ] 20:52:55 DEBUG } 20:52:55 DEBUG } 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383575" 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383575" 20:52:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383575" 20:52:55 INFO [http_cmd]: http status code OK 20:52:55 DEBUG --- status code --- 20:52:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:55 DEBUG --- http response --- 20:52:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:56 DEBUG ----- output ----- 20:52:56 DEBUG { 20:52:56 DEBUG "status": "success", 20:52:56 DEBUG "data": { 20:52:56 DEBUG "resultType": "vector", 20:52:56 DEBUG "result": [ 20:52:56 DEBUG { 20:52:56 DEBUG "metric": { 20:52:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG [...] 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 1713383575, 20:52:56 DEBUG "306.1333333333333" 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG } 20:52:56 INFO [http_cmd]: http status code OK 20:52:56 DEBUG --- status code --- 20:52:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:56 DEBUG --- http response --- 20:52:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:52:56 DEBUG ----- output ----- 20:52:56 DEBUG { 20:52:56 DEBUG "status": "success", 20:52:56 DEBUG "data": { 20:52:56 DEBUG "resultType": "vector", 20:52:56 DEBUG "result": [ 20:52:56 DEBUG { 20:52:56 DEBUG "metric": { 20:52:56 DEBUG "pod": "ds-cts-0" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG [...] 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG "pod": "ds-cts-2" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 1713383575, 20:52:56 DEBUG "0" 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG } 20:52:56 INFO [http_cmd]: http status code OK 20:52:56 DEBUG --- status code --- 20:52:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:56 DEBUG --- http response --- 20:52:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:56 DEBUG ----- output ----- 20:52:56 DEBUG { 20:52:56 DEBUG "status": "success", 20:52:56 DEBUG "data": { 20:52:56 DEBUG "resultType": "vector", 20:52:56 DEBUG "result": [ 20:52:56 DEBUG { 20:52:56 DEBUG "metric": { 20:52:56 DEBUG "pod": "ds-idrepo-1" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG [...] 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 1713383575, 20:52:56 DEBUG "0" 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG } 20:52:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383576" 20:52:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383576" 20:52:56 INFO [http_cmd]: http status code OK 20:52:56 DEBUG --- status code --- 20:52:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:56 DEBUG --- http response --- 20:52:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:56 DEBUG ----- output ----- 20:52:56 DEBUG { 20:52:56 DEBUG "status": "success", 20:52:56 DEBUG "data": { 20:52:56 DEBUG "resultType": "vector", 20:52:56 DEBUG "result": [ 20:52:56 DEBUG { 20:52:56 DEBUG "metric": { 20:52:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG [...] 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 1713383576, 20:52:56 DEBUG "253878" 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG } 20:52:56 INFO [http_cmd]: http status code OK 20:52:56 DEBUG --- status code --- 20:52:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:56 DEBUG --- http response --- 20:52:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:56 DEBUG ----- output ----- 20:52:56 DEBUG { 20:52:56 DEBUG "status": "success", 20:52:56 DEBUG "data": { 20:52:56 DEBUG "resultType": "vector", 20:52:56 DEBUG "result": [ 20:52:56 DEBUG { 20:52:56 DEBUG "metric": { 20:52:56 DEBUG "pod": "ds-idrepo-0" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG [...] 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG "pod": "ds-idrepo-2" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 1713383576, 20:52:56 DEBUG "0" 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG } 20:52:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383576" 20:52:56 INFO [http_cmd]: http status code OK 20:52:56 DEBUG --- status code --- 20:52:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:56 DEBUG --- http response --- 20:52:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:56 DEBUG ----- output ----- 20:52:56 DEBUG { 20:52:56 DEBUG "status": "success", 20:52:56 DEBUG "data": { 20:52:56 DEBUG "resultType": "vector", 20:52:56 DEBUG "result": [ 20:52:56 DEBUG { 20:52:56 DEBUG "metric": { 20:52:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG [...] 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 1713383576, 20:52:56 DEBUG "1.4886646542707862" 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG } 20:52:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383576" 20:52:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383576" 20:52:56 INFO [http_cmd]: http status code OK 20:52:56 DEBUG --- status code --- 20:52:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:56 DEBUG --- http response --- 20:52:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:52:56 DEBUG ----- output ----- 20:52:56 DEBUG { 20:52:56 DEBUG "status": "success", 20:52:56 DEBUG "data": { 20:52:56 DEBUG "resultType": "vector", 20:52:56 DEBUG "result": [ 20:52:56 DEBUG { 20:52:56 DEBUG "metric": { 20:52:56 DEBUG "pod": "ds-cts-0" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG [...] 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG "pod": "ds-cts-2" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 1713383576, 20:52:56 DEBUG "29.866666666666667" 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG } 20:52:56 INFO [http_cmd]: http status code OK 20:52:56 DEBUG --- status code --- 20:52:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:56 DEBUG --- http response --- 20:52:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:56 DEBUG ----- output ----- 20:52:56 DEBUG { 20:52:56 DEBUG "status": "success", 20:52:56 DEBUG "data": { 20:52:56 DEBUG "resultType": "vector", 20:52:56 DEBUG "result": [ 20:52:56 DEBUG { 20:52:56 DEBUG "metric": { 20:52:56 DEBUG "instance": "10.142.0.107:9100" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG [...] 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG "instance": "10.142.0.43:9100" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 1713383576, 20:52:56 DEBUG "16685782.866666665" 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG } 20:52:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383576" 20:52:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383576" 20:52:56 INFO [http_cmd]: http status code OK 20:52:56 DEBUG --- status code --- 20:52:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:56 DEBUG --- http response --- 20:52:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:56 DEBUG ----- output ----- 20:52:56 DEBUG { 20:52:56 DEBUG "status": "success", 20:52:56 DEBUG "data": { 20:52:56 DEBUG "resultType": "vector", 20:52:56 DEBUG "result": [ 20:52:56 DEBUG { 20:52:56 DEBUG "metric": { 20:52:56 DEBUG "pod": "ds-idrepo-0" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG [...] 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG "pod": "ds-idrepo-2" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 1713383576, 20:52:56 DEBUG "0" 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG } 20:52:56 INFO [http_cmd]: http status code OK 20:52:56 DEBUG --- status code --- 20:52:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:56 DEBUG --- http response --- 20:52:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:52:56 DEBUG ----- output ----- 20:52:56 DEBUG { 20:52:56 DEBUG "status": "success", 20:52:56 DEBUG "data": { 20:52:56 DEBUG "resultType": "vector", 20:52:56 DEBUG "result": [ 20:52:56 DEBUG { 20:52:56 DEBUG "metric": { 20:52:56 DEBUG "pod": "ds-idrepo-0" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG [...] 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG "pod": "ds-idrepo-2" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 1713383576, 20:52:56 DEBUG "0" 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG } 20:52:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383576" 20:52:56 INFO [http_cmd]: http status code OK 20:52:56 DEBUG --- status code --- 20:52:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:56 DEBUG --- http response --- 20:52:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 20:52:56 DEBUG ----- output ----- 20:52:56 DEBUG { 20:52:56 DEBUG "status": "success", 20:52:56 DEBUG "data": { 20:52:56 DEBUG "resultType": "vector", 20:52:56 DEBUG "result": [ 20:52:56 DEBUG { 20:52:56 DEBUG "metric": { 20:52:56 DEBUG "pod": "ds-idrepo-0" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG [...] 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG "pod": "idm-b85f9d978-txpwv" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 1713383576, 20:52:56 DEBUG "560.7886089813801" 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG } 20:52:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383576" 20:52:56 INFO [http_cmd]: http status code OK 20:52:56 DEBUG --- status code --- 20:52:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:52:56 DEBUG --- http response --- 20:52:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:52:56 DEBUG ----- output ----- 20:52:56 DEBUG { 20:52:56 DEBUG "status": "success", 20:52:56 DEBUG "data": { 20:52:56 DEBUG "resultType": "vector", 20:52:56 DEBUG "result": [ 20:52:56 DEBUG { 20:52:56 DEBUG "metric": { 20:52:56 DEBUG "instance": "10.142.0.107:9100" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG [...] 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG 20:52:56 DEBUG "instance": "10.142.0.43:9100" 20:52:56 DEBUG }, 20:52:56 DEBUG "value": [ 20:52:56 DEBUG 1713383576, 20:52:56 DEBUG "0.07099999999999992" 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG ] 20:52:56 DEBUG } 20:52:56 DEBUG } 20:52:56 INFO 20:52:56 INFO [loop_until]: kubectl --namespace=xlou top pods 20:52:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:52:56 INFO [loop_until]: OK (rc = 0) 20:52:56 DEBUG --- stdout --- 20:52:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 2779m 8889Mi am-b59c765b7-bgsxp 2280m 8712Mi am-b59c765b7-ch5q4 2066m 8742Mi ds-cts-0 6668m 7729Mi ds-cts-1 5152m 7579Mi ds-cts-2 6055m 7853Mi ds-idrepo-0 947m 3082Mi ds-idrepo-1 553m 2758Mi ds-idrepo-2 545m 2723Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1004Mi lodemon-db4967764-9zt9t 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1470m 461Mi 20:52:56 DEBUG --- stderr --- 20:52:56 DEBUG 20:52:57 INFO 20:52:57 INFO [loop_until]: kubectl --namespace=xlou top node 20:52:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:52:57 INFO [loop_until]: OK (rc = 0) 20:52:57 DEBUG --- stdout --- 20:52:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 5490Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2854m 9% 5960Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 5548m 18% 11330Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 2980Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 7521m 25% 10020Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 5001Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 5875m 19% 11506Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 8446m 28% 2897Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 4666m 15% 10564Mi 9% gke-xlou-cdm-ds-0f601751-j488 1554m 5% 5258Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 8155m 27% 11011Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 2333m 7% 5825Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 8989m 30% 9512Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 3451m 21% 4536Mi 7% 20:52:57 DEBUG --- stderr --- 20:52:57 DEBUG 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383635" 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383635" 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383635" 20:53:55 INFO [http_cmd]: http status code OK 20:53:55 DEBUG --- status code --- 20:53:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:55 DEBUG --- http response --- 20:53:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:55 DEBUG ----- output ----- 20:53:55 DEBUG { 20:53:55 DEBUG "status": "success", 20:53:55 DEBUG "data": { 20:53:55 DEBUG "resultType": "vector", 20:53:55 DEBUG "result": [ 20:53:55 DEBUG { 20:53:55 DEBUG "metric": { 20:53:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG [...] 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 1713383635, 20:53:55 DEBUG "1829.3666666666666" 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG } 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383635" 20:53:55 INFO [http_cmd]: http status code OK 20:53:55 INFO [http_cmd]: http status code OK 20:53:55 DEBUG --- status code --- 20:53:55 DEBUG --- status code --- 20:53:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:55 DEBUG --- http response --- 20:53:55 DEBUG --- http response --- 20:53:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:55 DEBUG ----- output ----- 20:53:55 DEBUG ----- output ----- 20:53:55 DEBUG { 20:53:55 DEBUG { 20:53:55 DEBUG "status": "success", 20:53:55 DEBUG "status": "success", 20:53:55 DEBUG "data": { 20:53:55 DEBUG "data": { 20:53:55 DEBUG "resultType": "vector", 20:53:55 DEBUG "resultType": "vector", 20:53:55 DEBUG "result": [ 20:53:55 DEBUG "result": [ 20:53:55 DEBUG { 20:53:55 DEBUG { 20:53:55 DEBUG "metric": { 20:53:55 DEBUG "metric": { 20:53:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:53:55 DEBUG "instance": "10.142.0.107:9100" 20:53:55 DEBUG }, 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG [...] 20:53:55 DEBUG [...] 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:53:55 DEBUG "instance": "10.142.0.43:9100" 20:53:55 DEBUG }, 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 1713383635, 20:53:55 DEBUG 1713383635, 20:53:55 DEBUG "1096.5666666666666" 20:53:55 DEBUG "0.05936666666666781" 20:53:55 DEBUG ] 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG } 20:53:55 DEBUG ] 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG } 20:53:55 DEBUG } 20:53:55 DEBUG } 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383635" 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383635" 20:53:55 INFO [http_cmd]: http status code OK 20:53:55 DEBUG --- status code --- 20:53:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:55 DEBUG --- http response --- 20:53:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:55 DEBUG ----- output ----- 20:53:55 DEBUG { 20:53:55 DEBUG "status": "success", 20:53:55 DEBUG "data": { 20:53:55 DEBUG "resultType": "vector", 20:53:55 DEBUG "result": [ 20:53:55 DEBUG { 20:53:55 DEBUG "metric": { 20:53:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG [...] 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 1713383635, 20:53:55 DEBUG "1096.5666666666666" 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG } 20:53:55 INFO [http_cmd]: http status code OK 20:53:55 DEBUG --- status code --- 20:53:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:55 DEBUG --- http response --- 20:53:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:53:55 DEBUG ----- output ----- 20:53:55 DEBUG { 20:53:55 DEBUG "status": "success", 20:53:55 DEBUG "data": { 20:53:55 DEBUG "resultType": "vector", 20:53:55 DEBUG "result": [ 20:53:55 DEBUG { 20:53:55 DEBUG "metric": { 20:53:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG [...] 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 1713383635, 20:53:55 DEBUG "65754183.33209427" 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG } 20:53:55 INFO [http_cmd]: http status code OK 20:53:55 DEBUG --- status code --- 20:53:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:55 DEBUG --- http response --- 20:53:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383635" 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383635" 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383635" 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383635" 20:53:55 INFO [http_cmd]: http status code OK 20:53:55 INFO [http_cmd]: http status code OK 20:53:55 DEBUG --- status code --- 20:53:55 DEBUG --- status code --- 20:53:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:55 DEBUG --- http response --- 20:53:55 DEBUG --- http response --- 20:53:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:55 DEBUG ----- output ----- 20:53:55 DEBUG { 20:53:55 DEBUG "status": "success", 20:53:55 DEBUG "data": { 20:53:55 DEBUG "resultType": "vector", 20:53:55 DEBUG "result": [ 20:53:55 DEBUG { 20:53:55 DEBUG "metric": { 20:53:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG [...] 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 1713383635, 20:53:55 DEBUG "0.012" 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG } 20:53:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:53:55 DEBUG ----- output ----- 20:53:55 DEBUG { 20:53:55 DEBUG "status": "success", 20:53:55 DEBUG "data": { 20:53:55 DEBUG "resultType": "vector", 20:53:55 DEBUG "result": [ 20:53:55 DEBUG { 20:53:55 DEBUG "metric": { 20:53:55 DEBUG "pod": "ds-cts-0" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG [...] 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG "pod": "ds-cts-2" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 1713383635, 20:53:55 DEBUG "0.00003532000000000001" 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG } 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383635" 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383635" 20:53:55 INFO [http_cmd]: http status code OK 20:53:55 DEBUG --- status code --- 20:53:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:55 DEBUG --- http response --- 20:53:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:53:55 INFO [http_cmd]: http status code OK 20:53:55 DEBUG ----- output ----- 20:53:55 DEBUG --- status code --- 20:53:55 DEBUG { 20:53:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:55 DEBUG "status": "success", 20:53:55 DEBUG --- http response --- 20:53:55 DEBUG "data": { 20:53:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:55 DEBUG "resultType": "vector", 20:53:55 DEBUG ----- output ----- 20:53:55 DEBUG "result": [ 20:53:55 DEBUG { 20:53:55 DEBUG { 20:53:55 DEBUG "status": "success", 20:53:55 DEBUG "metric": { 20:53:55 DEBUG "data": { 20:53:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:53:55 DEBUG "resultType": "vector", 20:53:55 DEBUG }, 20:53:55 DEBUG "result": [ 20:53:55 DEBUG "value": [ 20:53:55 DEBUG { 20:53:55 DEBUG 20:53:55 DEBUG "metric": { 20:53:55 DEBUG 20:53:55 DEBUG "instance": "10.142.0.107:9100" 20:53:55 DEBUG 20:53:55 DEBUG }, 20:53:55 DEBUG [...] 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:53:55 DEBUG [...] 20:53:55 DEBUG }, 20:53:55 DEBUG 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 20:53:55 DEBUG 1713383635, 20:53:55 DEBUG 20:53:55 DEBUG "7.892244141476463" 20:53:55 DEBUG "instance": "10.142.0.43:9100" 20:53:55 DEBUG ] 20:53:55 DEBUG }, 20:53:55 DEBUG } 20:53:55 DEBUG "value": [ 20:53:55 DEBUG ] 20:53:55 DEBUG 1713383635, 20:53:55 DEBUG } 20:53:55 DEBUG "45390736.13333333" 20:53:55 DEBUG } 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG } 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383635" 20:53:55 ERROR [http_cmd]: ERROR 20:53:55 INFO [http_cmd]: http status code OK 20:53:55 ERROR --- status code --- 20:53:55 DEBUG --- status code --- 20:53:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:53:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:55 ERROR --- http response --- 20:53:55 DEBUG --- http response --- 20:53:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:53:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:53:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:53:55 DEBUG ----- output ----- 20:53:55 DEBUG { 20:53:55 DEBUG "status": "success", 20:53:55 DEBUG "data": { 20:53:55 DEBUG "resultType": "vector", 20:53:55 DEBUG "result": [ 20:53:55 DEBUG { 20:53:55 DEBUG "metric": { 20:53:55 DEBUG "pod": "ds-idrepo-0" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG [...] 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG "pod": "ds-cts-2" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 1713383635, 20:53:55 DEBUG "0" 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG } 20:53:55 INFO [http_cmd]: http status code OK 20:53:55 DEBUG --- status code --- 20:53:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:55 DEBUG --- http response --- 20:53:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:55 DEBUG ----- output ----- 20:53:55 DEBUG { 20:53:55 DEBUG "status": "success", 20:53:55 DEBUG "data": { 20:53:55 DEBUG "resultType": "vector", 20:53:55 DEBUG "result": [ 20:53:55 DEBUG { 20:53:55 DEBUG "metric": { 20:53:55 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG [...] 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 1713383635, 20:53:55 DEBUG "2810183680" 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG ] 20:53:55 DEBUG } 20:53:55 DEBUG } 20:53:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383635" 20:53:55 INFO [http_cmd]: http status code OK 20:53:55 DEBUG --- status code --- 20:53:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:55 DEBUG --- http response --- 20:53:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:55 DEBUG ----- output ----- 20:53:55 DEBUG { 20:53:55 DEBUG "status": "success", 20:53:55 DEBUG "data": { 20:53:55 DEBUG "resultType": "vector", 20:53:55 DEBUG "result": [ 20:53:55 DEBUG { 20:53:55 DEBUG "metric": { 20:53:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG [...] 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG 20:53:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:53:55 DEBUG }, 20:53:55 DEBUG "value": [ 20:53:56 DEBUG 1713383635, 20:53:56 DEBUG "0" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383636" 20:53:56 INFO [http_cmd]: http status code OK 20:53:56 DEBUG --- status code --- 20:53:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:56 DEBUG --- http response --- 20:53:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:53:56 DEBUG ----- output ----- 20:53:56 DEBUG { 20:53:56 DEBUG "status": "success", 20:53:56 DEBUG "data": { 20:53:56 DEBUG "resultType": "vector", 20:53:56 DEBUG "result": [ 20:53:56 DEBUG { 20:53:56 DEBUG "metric": { 20:53:56 DEBUG "pod": "ds-cts-0" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG [...] 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG "pod": "ds-cts-2" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 1713383636, 20:53:56 DEBUG "0" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383636" 20:53:56 INFO [http_cmd]: http status code OK 20:53:56 DEBUG --- status code --- 20:53:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:56 DEBUG --- http response --- 20:53:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:56 DEBUG ----- output ----- 20:53:56 DEBUG { 20:53:56 DEBUG "status": "success", 20:53:56 DEBUG "data": { 20:53:56 DEBUG "resultType": "vector", 20:53:56 DEBUG "result": [ 20:53:56 DEBUG { 20:53:56 DEBUG "metric": { 20:53:56 DEBUG "instance": "10.142.0.107:9100" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG [...] 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG "instance": "10.142.0.43:9100" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 1713383636, 20:53:56 DEBUG "150295493632" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383636" 20:53:56 INFO [http_cmd]: http status code OK 20:53:56 DEBUG --- status code --- 20:53:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:56 DEBUG --- http response --- 20:53:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:56 DEBUG ----- output ----- 20:53:56 DEBUG { 20:53:56 DEBUG "status": "success", 20:53:56 DEBUG "data": { 20:53:56 DEBUG "resultType": "vector", 20:53:56 DEBUG "result": [ 20:53:56 DEBUG { 20:53:56 DEBUG "metric": { 20:53:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG [...] 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 1713383636, 20:53:56 DEBUG "1096.5666666666666" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383636" 20:53:56 INFO [http_cmd]: http status code OK 20:53:56 DEBUG --- status code --- 20:53:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:56 DEBUG --- http response --- 20:53:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:56 DEBUG ----- output ----- 20:53:56 DEBUG { 20:53:56 DEBUG "status": "success", 20:53:56 DEBUG "data": { 20:53:56 DEBUG "resultType": "vector", 20:53:56 DEBUG "result": [ 20:53:56 DEBUG { 20:53:56 DEBUG "metric": { 20:53:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG [...] 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 1713383636, 20:53:56 DEBUG "0" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383636" 20:53:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383636" 20:53:56 INFO [http_cmd]: http status code OK 20:53:56 DEBUG --- status code --- 20:53:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:56 DEBUG --- http response --- 20:53:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:56 DEBUG ----- output ----- 20:53:56 DEBUG { 20:53:56 DEBUG "status": "success", 20:53:56 DEBUG "data": { 20:53:56 DEBUG "resultType": "vector", 20:53:56 DEBUG "result": [ 20:53:56 DEBUG { 20:53:56 DEBUG "metric": { 20:53:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG [...] 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 1713383636, 20:53:56 DEBUG "215227" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:56 INFO [http_cmd]: http status code OK 20:53:56 DEBUG --- status code --- 20:53:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:56 DEBUG --- http response --- 20:53:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:56 DEBUG ----- output ----- 20:53:56 DEBUG { 20:53:56 DEBUG "status": "success", 20:53:56 DEBUG "data": { 20:53:56 DEBUG "resultType": "vector", 20:53:56 DEBUG "result": [ 20:53:56 DEBUG { 20:53:56 DEBUG "metric": { 20:53:56 DEBUG "pod": "ds-idrepo-0" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG [...] 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG "pod": "ds-idrepo-2" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 1713383636, 20:53:56 DEBUG "0" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383636" 20:53:56 INFO [http_cmd]: http status code OK 20:53:56 DEBUG --- status code --- 20:53:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:56 DEBUG --- http response --- 20:53:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:56 DEBUG ----- output ----- 20:53:56 DEBUG { 20:53:56 DEBUG "status": "success", 20:53:56 DEBUG "data": { 20:53:56 DEBUG "resultType": "vector", 20:53:56 DEBUG "result": [ 20:53:56 DEBUG { 20:53:56 DEBUG "metric": { 20:53:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG [...] 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 1713383636, 20:53:56 DEBUG "7.771141077894664" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383636" 20:53:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383636" 20:53:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383636" 20:53:56 INFO [http_cmd]: http status code OK 20:53:56 DEBUG --- status code --- 20:53:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:56 DEBUG --- http response --- 20:53:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:53:56 DEBUG ----- output ----- 20:53:56 DEBUG { 20:53:56 DEBUG "status": "success", 20:53:56 DEBUG "data": { 20:53:56 DEBUG "resultType": "vector", 20:53:56 DEBUG "result": [ 20:53:56 DEBUG { 20:53:56 DEBUG "metric": { 20:53:56 DEBUG "pod": "ds-cts-0" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG [...] 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG "pod": "ds-cts-2" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 1713383636, 20:53:56 DEBUG "30" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:56 INFO [http_cmd]: http status code OK 20:53:56 DEBUG --- status code --- 20:53:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:56 DEBUG --- http response --- 20:53:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:56 DEBUG ----- output ----- 20:53:56 DEBUG { 20:53:56 DEBUG "status": "success", 20:53:56 DEBUG "data": { 20:53:56 DEBUG "resultType": "vector", 20:53:56 DEBUG "result": [ 20:53:56 DEBUG { 20:53:56 DEBUG "metric": { 20:53:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG [...] 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 1713383636, 20:53:56 DEBUG "0" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:56 INFO [http_cmd]: http status code OK 20:53:56 DEBUG --- status code --- 20:53:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:56 DEBUG --- http response --- 20:53:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:56 DEBUG ----- output ----- 20:53:56 DEBUG { 20:53:56 DEBUG "status": "success", 20:53:56 DEBUG "data": { 20:53:56 DEBUG "resultType": "vector", 20:53:56 DEBUG "result": [ 20:53:56 DEBUG { 20:53:56 DEBUG "metric": { 20:53:56 DEBUG "instance": "10.142.0.107:9100" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG [...] 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG "instance": "10.142.0.43:9100" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 1713383636, 20:53:56 DEBUG "45382943.03333333" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383636" 20:53:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383636" 20:53:56 INFO [http_cmd]: http status code OK 20:53:56 DEBUG --- status code --- 20:53:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:56 DEBUG --- http response --- 20:53:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:56 DEBUG ----- output ----- 20:53:56 DEBUG { 20:53:56 DEBUG "status": "success", 20:53:56 DEBUG "data": { 20:53:56 DEBUG "resultType": "vector", 20:53:56 DEBUG "result": [ 20:53:56 DEBUG { 20:53:56 DEBUG "metric": { 20:53:56 DEBUG "pod": "ds-idrepo-0" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG [...] 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG "pod": "ds-idrepo-2" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 1713383636, 20:53:56 DEBUG "0" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:56 INFO [http_cmd]: http status code OK 20:53:56 DEBUG --- status code --- 20:53:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:56 DEBUG --- http response --- 20:53:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:53:56 DEBUG ----- output ----- 20:53:56 DEBUG { 20:53:56 DEBUG "status": "success", 20:53:56 DEBUG "data": { 20:53:56 DEBUG "resultType": "vector", 20:53:56 DEBUG "result": [ 20:53:56 DEBUG { 20:53:56 DEBUG "metric": { 20:53:56 DEBUG "pod": "ds-idrepo-0" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG [...] 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG "pod": "ds-idrepo-2" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 1713383636, 20:53:56 DEBUG "0" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383636" 20:53:56 INFO [http_cmd]: http status code OK 20:53:56 DEBUG --- status code --- 20:53:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:53:56 DEBUG --- http response --- 20:53:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:53:56 DEBUG ----- output ----- 20:53:56 DEBUG { 20:53:56 DEBUG "status": "success", 20:53:56 DEBUG "data": { 20:53:56 DEBUG "resultType": "vector", 20:53:56 DEBUG "result": [ 20:53:56 DEBUG { 20:53:56 DEBUG "metric": { 20:53:56 DEBUG "instance": "10.142.0.107:9100" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG [...] 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG 20:53:56 DEBUG "instance": "10.142.0.43:9100" 20:53:56 DEBUG }, 20:53:56 DEBUG "value": [ 20:53:56 DEBUG 1713383636, 20:53:56 DEBUG "0.03500000000000013" 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG ] 20:53:56 DEBUG } 20:53:56 DEBUG } 20:53:57 INFO 20:53:57 INFO [loop_until]: kubectl --namespace=xlou top pods 20:53:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:53:57 INFO [loop_until]: OK (rc = 0) 20:53:57 DEBUG --- stdout --- 20:53:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 8525m 8457Mi am-b59c765b7-bgsxp 7067m 8650Mi am-b59c765b7-ch5q4 7830m 8764Mi ds-cts-0 9477m 7811Mi ds-cts-1 8724m 7602Mi ds-cts-2 9470m 7869Mi ds-idrepo-0 3299m 3081Mi ds-idrepo-1 3247m 2766Mi ds-idrepo-2 3310m 2723Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 966Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7872m 3721Mi 20:53:57 DEBUG --- stderr --- 20:53:57 DEBUG 20:53:57 INFO 20:53:57 INFO [loop_until]: kubectl --namespace=xlou top node 20:53:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:53:57 INFO [loop_until]: OK (rc = 0) 20:53:57 DEBUG --- stdout --- 20:53:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 5486Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3399m 11% 5959Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7657m 25% 11350Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 74m 0% 2982Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9152m 30% 10021Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 4991Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 7888m 26% 11079Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 16557m 55% 3700Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 7831m 26% 10490Mi 9% gke-xlou-cdm-ds-0f601751-j488 3421m 11% 5266Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 8680m 29% 11025Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 3510m 11% 5843Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 9611m 32% 9531Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 8165m 51% 7302Mi 12% 20:53:57 DEBUG --- stderr --- 20:53:57 DEBUG 20:54:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383695" 20:54:55 INFO [http_cmd]: http status code OK 20:54:55 DEBUG --- status code --- 20:54:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:55 DEBUG --- http response --- 20:54:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:55 DEBUG ----- output ----- 20:54:55 DEBUG { 20:54:55 DEBUG "status": "success", 20:54:55 DEBUG "data": { 20:54:55 DEBUG "resultType": "vector", 20:54:55 DEBUG "result": [ 20:54:55 DEBUG { 20:54:55 DEBUG "metric": { 20:54:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG [...] 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 1713383695, 20:54:55 DEBUG "1823.9666666666667" 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG } 20:54:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383695" 20:54:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383695" 20:54:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383695" 20:54:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383695" 20:54:55 INFO [http_cmd]: http status code OK 20:54:55 INFO [http_cmd]: http status code OK 20:54:55 DEBUG --- status code --- 20:54:55 DEBUG --- status code --- 20:54:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:55 DEBUG --- http response --- 20:54:55 DEBUG --- http response --- 20:54:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:55 DEBUG ----- output ----- 20:54:55 DEBUG { 20:54:55 DEBUG "status": "success", 20:54:55 DEBUG "data": { 20:54:55 DEBUG "resultType": "vector", 20:54:55 DEBUG "result": [ 20:54:55 DEBUG { 20:54:55 DEBUG "metric": { 20:54:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG [...] 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 1713383695, 20:54:55 DEBUG "1054.1" 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG } 20:54:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:55 INFO [http_cmd]: http status code OK 20:54:55 DEBUG ----- output ----- 20:54:55 DEBUG --- status code --- 20:54:55 DEBUG { 20:54:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:55 DEBUG "status": "success", 20:54:55 DEBUG --- http response --- 20:54:55 DEBUG "data": { 20:54:55 DEBUG "resultType": "vector", 20:54:55 DEBUG "result": [ 20:54:55 DEBUG { 20:54:55 DEBUG "metric": { 20:54:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG [...] 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 1713383695, 20:54:55 DEBUG "1054.1" 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG } 20:54:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:55 DEBUG ----- output ----- 20:54:55 DEBUG { 20:54:55 DEBUG "status": "success", 20:54:55 DEBUG "data": { 20:54:55 DEBUG "resultType": "vector", 20:54:55 DEBUG "result": [ 20:54:55 DEBUG { 20:54:55 DEBUG "metric": { 20:54:55 DEBUG "instance": "10.142.0.107:9100" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG [...] 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG "instance": "10.142.0.43:9100" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 1713383695, 20:54:55 DEBUG "0.049466666666666964" 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG } 20:54:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383695" 20:54:55 INFO [http_cmd]: http status code OK 20:54:55 DEBUG --- status code --- 20:54:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:55 DEBUG --- http response --- 20:54:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:54:55 DEBUG ----- output ----- 20:54:55 DEBUG { 20:54:55 DEBUG "status": "success", 20:54:55 DEBUG "data": { 20:54:55 DEBUG "resultType": "vector", 20:54:55 DEBUG "result": [ 20:54:55 DEBUG { 20:54:55 DEBUG "metric": { 20:54:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG [...] 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG "pod": "ds-cts-0" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 1713383695, 20:54:55 DEBUG "4.679996109475234" 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG } 20:54:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383695" 20:54:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383695" 20:54:55 INFO [http_cmd]: http status code OK 20:54:55 DEBUG --- status code --- 20:54:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:55 DEBUG --- http response --- 20:54:55 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:54:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383695" 20:54:55 ERROR [http_cmd]: ERROR 20:54:55 INFO [http_cmd]: http status code OK 20:54:55 ERROR --- status code --- 20:54:55 DEBUG --- status code --- 20:54:55 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:54:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:55 ERROR --- http response --- 20:54:55 DEBUG --- http response --- 20:54:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383695" 20:54:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383695" 20:54:55 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:54:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:54:55 DEBUG ----- output ----- 20:54:55 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:54:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383695" 20:54:55 DEBUG { 20:54:55 DEBUG "status": "success", 20:54:55 DEBUG "data": { 20:54:55 DEBUG "resultType": "vector", 20:54:55 DEBUG "result": [ 20:54:55 DEBUG { 20:54:55 DEBUG "metric": { 20:54:55 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG [...] 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG "pod": "ds-cts-0" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 1713383695, 20:54:55 DEBUG "490008.2336249576" 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG } 20:54:55 INFO [http_cmd]: http status code OK 20:54:55 DEBUG --- status code --- 20:54:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:55 DEBUG --- http response --- 20:54:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:55 DEBUG ----- output ----- 20:54:55 DEBUG { 20:54:55 DEBUG "status": "success", 20:54:55 DEBUG "data": { 20:54:55 DEBUG "resultType": "vector", 20:54:55 DEBUG "result": [ 20:54:55 DEBUG { 20:54:55 DEBUG "metric": { 20:54:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG [...] 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 1713383695, 20:54:55 DEBUG "0.012" 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG } 20:54:55 INFO [http_cmd]: http status code OK 20:54:55 DEBUG --- status code --- 20:54:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:55 DEBUG --- http response --- 20:54:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:55 DEBUG ----- output ----- 20:54:55 DEBUG { 20:54:55 DEBUG "status": "success", 20:54:55 DEBUG "data": { 20:54:55 DEBUG "resultType": "vector", 20:54:55 DEBUG "result": [ 20:54:55 DEBUG { 20:54:55 DEBUG "metric": { 20:54:55 DEBUG "instance": "10.142.0.107:9100" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG [...] 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG "instance": "10.142.0.43:9100" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 1713383695, 20:54:55 DEBUG "40821489.16666667" 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG } 20:54:55 INFO [http_cmd]: http status code OK 20:54:55 DEBUG --- status code --- 20:54:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:55 DEBUG --- http response --- 20:54:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:54:55 DEBUG ----- output ----- 20:54:55 DEBUG { 20:54:55 DEBUG "status": "success", 20:54:55 DEBUG "data": { 20:54:55 DEBUG "resultType": "vector", 20:54:55 DEBUG "result": [ 20:54:55 DEBUG { 20:54:55 DEBUG "metric": { 20:54:55 DEBUG "pod": "ds-idrepo-0" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG [...] 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG "pod": "ds-cts-2" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 1713383695, 20:54:55 DEBUG "0" 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG ] 20:54:55 DEBUG } 20:54:55 DEBUG } 20:54:55 INFO [http_cmd]: http status code OK 20:54:55 DEBUG --- status code --- 20:54:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:55 DEBUG --- http response --- 20:54:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:54:55 DEBUG ----- output ----- 20:54:55 DEBUG { 20:54:55 DEBUG "status": "success", 20:54:55 DEBUG "data": { 20:54:55 DEBUG "resultType": "vector", 20:54:55 DEBUG "result": [ 20:54:55 DEBUG { 20:54:55 DEBUG "metric": { 20:54:55 DEBUG "pod": "ds-cts-0" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG [...] 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG 20:54:55 DEBUG "pod": "ds-cts-2" 20:54:55 DEBUG }, 20:54:55 DEBUG "value": [ 20:54:56 DEBUG 1713383695, 20:54:56 DEBUG "0" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383696" 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "5270470656" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383696" 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "0" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383696" 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "instance": "10.142.0.107:9100" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "instance": "10.142.0.43:9100" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "152307987456" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383696" 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383696" 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383696" 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "pod": "ds-cts-0" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "0" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "pod": "ds-cts-0" 20:54:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:54:56 DEBUG }, 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "pod": "ds-cts-2" 20:54:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:54:56 DEBUG }, 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "0" 20:54:56 DEBUG "1054.1" 20:54:56 DEBUG ] 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383696" 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383696" 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "168527" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "pod": "ds-idrepo-0" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "pod": "ds-idrepo-2" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "0" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383696" 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "pod": "ds-cts-0" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "pod": "ds-cts-2" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "29.933333333333334" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383696" 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "8.015634293673225" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383696" 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383696" 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "instance": "10.142.0.107:9100" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "instance": "10.142.0.43:9100" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "40813248.766666666" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "pod": "ds-cts-0" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "16218213.577049848" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383696" 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383696" 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "pod": "ds-idrepo-0" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "pod": "ds-idrepo-2" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "0" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "pod": "ds-idrepo-0" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "pod": "ds-idrepo-2" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "0" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383696" 20:54:56 INFO [http_cmd]: http status code OK 20:54:56 DEBUG --- status code --- 20:54:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:54:56 DEBUG --- http response --- 20:54:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:54:56 DEBUG ----- output ----- 20:54:56 DEBUG { 20:54:56 DEBUG "status": "success", 20:54:56 DEBUG "data": { 20:54:56 DEBUG "resultType": "vector", 20:54:56 DEBUG "result": [ 20:54:56 DEBUG { 20:54:56 DEBUG "metric": { 20:54:56 DEBUG "instance": "10.142.0.107:9100" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG [...] 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG 20:54:56 DEBUG "instance": "10.142.0.43:9100" 20:54:56 DEBUG }, 20:54:56 DEBUG "value": [ 20:54:56 DEBUG 1713383696, 20:54:56 DEBUG "0.03500000000000006" 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG ] 20:54:56 DEBUG } 20:54:56 DEBUG } 20:54:57 INFO 20:54:57 INFO [loop_until]: kubectl --namespace=xlou top pods 20:54:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:54:57 INFO [loop_until]: OK (rc = 0) 20:54:57 DEBUG --- stdout --- 20:54:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 7632m 8459Mi am-b59c765b7-bgsxp 6789m 8649Mi am-b59c765b7-ch5q4 7090m 8307Mi ds-cts-0 7206m 7812Mi ds-cts-1 7787m 7594Mi ds-cts-2 7542m 7861Mi ds-idrepo-0 3288m 3081Mi ds-idrepo-1 3078m 2766Mi ds-idrepo-2 3049m 2723Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7956m 6216Mi 20:54:57 DEBUG --- stderr --- 20:54:57 DEBUG 20:54:57 INFO 20:54:57 INFO [loop_until]: kubectl --namespace=xlou top node 20:54:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:54:58 INFO [loop_until]: OK (rc = 0) 20:54:58 DEBUG --- stdout --- 20:54:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 5494Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3078m 10% 5960Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7041m 23% 10887Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 2987Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8191m 27% 10017Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5001Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 7229m 24% 11073Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 16147m 54% 3693Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 7054m 23% 10493Mi 9% gke-xlou-cdm-ds-0f601751-j488 3316m 11% 5269Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 6558m 21% 11018Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 3344m 11% 5842Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 7156m 23% 9539Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 7898m 49% 9602Mi 16% 20:54:58 DEBUG --- stderr --- 20:54:58 DEBUG 20:55:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383755" 20:55:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383755" 20:55:55 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383755" 20:55:55 INFO [http_cmd]: http status code OK 20:55:55 DEBUG --- status code --- 20:55:55 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:55 DEBUG --- http response --- 20:55:55 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:55 DEBUG ----- output ----- 20:55:55 DEBUG { 20:55:55 DEBUG "status": "success", 20:55:55 DEBUG "data": { 20:55:55 DEBUG "resultType": "vector", 20:55:55 DEBUG "result": [ 20:55:55 DEBUG { 20:55:55 DEBUG "metric": { 20:55:55 DEBUG "pod": "am-b59c765b7-ch5q4" 20:55:55 DEBUG }, 20:55:55 DEBUG "value": [ 20:55:55 DEBUG 20:55:55 DEBUG 20:55:55 DEBUG 20:55:55 DEBUG [...] 20:55:55 DEBUG 20:55:55 DEBUG 20:55:55 DEBUG 20:55:55 DEBUG "pod": "am-b59c765b7-bgsxp" 20:55:55 DEBUG }, 20:55:55 DEBUG "value": [ 20:55:55 DEBUG 1713383755, 20:55:55 DEBUG "306.1666666666667" 20:55:55 DEBUG ] 20:55:55 DEBUG } 20:55:55 DEBUG ] 20:55:55 DEBUG } 20:55:55 DEBUG } 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "instance": "10.142.0.107:9100" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "instance": "10.142.0.43:9100" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383755, 20:55:56 DEBUG "0.029199999999999213" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "ds-idrepo-1" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383755, 20:55:56 DEBUG "0.0000256364922206537" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "1015.0666666666667" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "1015.0666666666666" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383756" 20:55:56 ERROR [http_cmd]: ERROR 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 ERROR --- status code --- 20:55:56 DEBUG --- status code --- 20:55:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 ERROR --- http response --- 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:55:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "ds-cts-0" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "ds-cts-2" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "0" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "ds-idrepo-1" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "0" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "ds-idrepo-0" 20:55:56 DEBUG }, 20:55:56 DEBUG "instance": "10.142.0.107:9100" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:55:56 DEBUG 20:55:56 DEBUG "pod": "ds-cts-2" 20:55:56 DEBUG }, 20:55:56 DEBUG "instance": "10.142.0.43:9100" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "0.012" 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "0" 20:55:56 DEBUG ] 20:55:56 DEBUG "28235358.4" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "7398047744" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "0" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383756" 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "instance": "10.142.0.107:9100" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "instance": "10.142.0.43:9100" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "153853033472" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "ds-idrepo-1" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "0" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "1015.1333333333333" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "ds-cts-0" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "ds-cts-2" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "0" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "ds-cts-0" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "ds-cts-2" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "30" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "196976" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "ds-idrepo-0" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "ds-idrepo-2" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "0" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "8.024248485921774" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383756" 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "instance": "10.142.0.107:9100" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "instance": "10.142.0.43:9100" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "28227289.833333332" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "ds-idrepo-1" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "0" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "ds-idrepo-0" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "ds-idrepo-2" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "0" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "pod": "ds-idrepo-0" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "pod": "ds-idrepo-2" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "0" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383756" 20:55:56 INFO [http_cmd]: http status code OK 20:55:56 DEBUG --- status code --- 20:55:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:55:56 DEBUG --- http response --- 20:55:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:55:56 DEBUG ----- output ----- 20:55:56 DEBUG { 20:55:56 DEBUG "status": "success", 20:55:56 DEBUG "data": { 20:55:56 DEBUG "resultType": "vector", 20:55:56 DEBUG "result": [ 20:55:56 DEBUG { 20:55:56 DEBUG "metric": { 20:55:56 DEBUG "instance": "10.142.0.107:9100" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG [...] 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG 20:55:56 DEBUG "instance": "10.142.0.43:9100" 20:55:56 DEBUG }, 20:55:56 DEBUG "value": [ 20:55:56 DEBUG 1713383756, 20:55:56 DEBUG "0.019999999999999955" 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG ] 20:55:56 DEBUG } 20:55:56 DEBUG } 20:55:57 INFO 20:55:57 INFO [loop_until]: kubectl --namespace=xlou top pods 20:55:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:55:57 INFO [loop_until]: OK (rc = 0) 20:55:57 DEBUG --- stdout --- 20:55:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 5600m 8456Mi am-b59c765b7-bgsxp 5624m 8634Mi am-b59c765b7-ch5q4 5917m 8303Mi ds-cts-0 5854m 7869Mi ds-cts-1 4587m 7606Mi ds-cts-2 5360m 7900Mi ds-idrepo-0 2752m 3082Mi ds-idrepo-1 2716m 2767Mi ds-idrepo-2 2602m 2723Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7951m 7394Mi 20:55:57 DEBUG --- stderr --- 20:55:57 DEBUG 20:55:58 INFO 20:55:58 INFO [loop_until]: kubectl --namespace=xlou top node 20:55:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:55:58 INFO [loop_until]: OK (rc = 0) 20:55:58 DEBUG --- stdout --- 20:55:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 5488Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2957m 9% 5958Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6162m 20% 10882Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 2986Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 4570m 15% 10052Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5012Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 6355m 21% 11066Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 15356m 51% 3696Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 5763m 19% 10484Mi 9% gke-xlou-cdm-ds-0f601751-j488 2799m 9% 5273Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 5359m 17% 11037Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 3027m 10% 5821Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 5940m 19% 9566Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 8265m 52% 10557Mi 18% 20:55:58 DEBUG --- stderr --- 20:55:58 DEBUG 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "ds-idrepo-1" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "8.002641830229418" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "1096.5666666666666" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383816" 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "988.6666666666666" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "instance": "10.142.0.107:9100" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "instance": "10.142.0.43:9100" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "0.039666666666666586" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "988.6666666666666" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "ds-idrepo-1" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "13506035.961272476" 20:56:56 DEBUG ] 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383816" 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG { 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "ds-idrepo-0" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "ds-cts-2" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "0" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "instance": "10.142.0.107:9100" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "instance": "10.142.0.43:9100" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "35604380.3" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "ds-cts-0" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "ds-cts-2" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "0.000018215555555555553" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "8212598784" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 ERROR [http_cmd]: ERROR 20:56:56 ERROR --- status code --- 20:56:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:56:56 ERROR --- http response --- 20:56:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:56:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "0.012" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "ds-idrepo-1" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "0" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "0" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "ds-cts-0" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "ds-cts-2" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "0" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "instance": "10.142.0.107:9100" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "instance": "10.142.0.43:9100" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "155641914368" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "988.6666666666666" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "ds-cts-0" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "ds-cts-2" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "29.933333333333334" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "186869.5" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "ds-idrepo-0" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "ds-idrepo-2" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "0" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383816" 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "8.0323728098022" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "instance": "10.142.0.107:9100" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "instance": "10.142.0.43:9100" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "35597124.9" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "ds-idrepo-0" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "ds-idrepo-2" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "0" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "ds-idrepo-1" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "0" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "pod": "ds-idrepo-0" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "pod": "ds-idrepo-2" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "0" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383816" 20:56:56 INFO [http_cmd]: http status code OK 20:56:56 DEBUG --- status code --- 20:56:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:56:56 DEBUG --- http response --- 20:56:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:56:56 DEBUG ----- output ----- 20:56:56 DEBUG { 20:56:56 DEBUG "status": "success", 20:56:56 DEBUG "data": { 20:56:56 DEBUG "resultType": "vector", 20:56:56 DEBUG "result": [ 20:56:56 DEBUG { 20:56:56 DEBUG "metric": { 20:56:56 DEBUG "instance": "10.142.0.107:9100" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG [...] 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG 20:56:56 DEBUG "instance": "10.142.0.43:9100" 20:56:56 DEBUG }, 20:56:56 DEBUG "value": [ 20:56:56 DEBUG 1713383816, 20:56:56 DEBUG "0.028333333333333197" 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG ] 20:56:56 DEBUG } 20:56:56 DEBUG } 20:56:57 INFO 20:56:57 INFO [loop_until]: kubectl --namespace=xlou top pods 20:56:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:56:57 INFO [loop_until]: OK (rc = 0) 20:56:57 DEBUG --- stdout --- 20:56:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 6429m 8458Mi am-b59c765b7-bgsxp 5840m 8634Mi am-b59c765b7-ch5q4 6161m 8308Mi ds-cts-0 6069m 7820Mi ds-cts-1 5843m 7600Mi ds-cts-2 6077m 7907Mi ds-idrepo-0 2759m 3082Mi ds-idrepo-1 2849m 2766Mi ds-idrepo-2 2647m 2723Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7300m 16Mi 20:56:57 DEBUG --- stderr --- 20:56:57 DEBUG 20:56:58 INFO 20:56:58 INFO [loop_until]: kubectl --namespace=xlou top node 20:56:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:56:58 INFO [loop_until]: OK (rc = 0) 20:56:58 DEBUG --- stdout --- 20:56:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 5488Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2641m 8% 5960Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6158m 20% 10895Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 2987Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 5962m 19% 10032Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5025Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 6379m 21% 11072Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 14038m 47% 3709Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 5971m 20% 10476Mi 9% gke-xlou-cdm-ds-0f601751-j488 2768m 9% 5268Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 5561m 18% 11027Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 2888m 9% 5836Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 5987m 20% 9564Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 8245m 51% 10978Mi 18% 20:56:58 DEBUG --- stderr --- 20:56:58 DEBUG 20:57:20 WARNING File /tmp/lodemon_data-20240417_201944/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [17/Apr/2024 20:57:20] "GET /monitoring/average?start_time=20240417_195408&stop_time=20240417_195519 HTTP/1.1" 200 - 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383876" 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:57:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:57:56 DEBUG }, 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:57:56 DEBUG "pod": "idm-b85f9d978-txpwv" 20:57:56 DEBUG }, 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "1054.2" 20:57:56 DEBUG "0" 20:57:56 DEBUG ] 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "instance": "10.142.0.107:9100" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "instance": "10.142.0.43:9100" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0.03733333333333538" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "idm-b85f9d978-txpwv" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383876" 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383876" 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "instance": "10.142.0.107:9100" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "instance": "10.142.0.43:9100" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "7706572.433333334" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 ERROR [http_cmd]: ERROR 20:57:56 ERROR --- status code --- 20:57:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:57:56 ERROR --- http response --- 20:57:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:57:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383876" 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0.012" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "ds-idrepo-0" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "ds-cts-2" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "ds-cts-0" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "ds-cts-2" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0.000033333333333333335" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "115122176" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "ds-idrepo-1" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "instance": "10.142.0.107:9100" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "instance": "10.142.0.43:9100" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "156784907264" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "ds-cts-0" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "ds-cts-2" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "147528.5" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "ds-cts-0" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "ds-cts-2" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "29.933333333333334" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "ds-idrepo-0" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "ds-idrepo-2" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383876" 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0.024446564127100563" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "instance": "10.142.0.107:9100" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "instance": "10.142.0.43:9100" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "7689458.833333334" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "ds-idrepo-0" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "ds-idrepo-2" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "ds-idrepo-1" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "5278.524931434425" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "pod": "ds-idrepo-0" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "pod": "ds-idrepo-2" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383876" 20:57:56 INFO [http_cmd]: http status code OK 20:57:56 DEBUG --- status code --- 20:57:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:57:56 DEBUG --- http response --- 20:57:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:57:56 DEBUG ----- output ----- 20:57:56 DEBUG { 20:57:56 DEBUG "status": "success", 20:57:56 DEBUG "data": { 20:57:56 DEBUG "resultType": "vector", 20:57:56 DEBUG "result": [ 20:57:56 DEBUG { 20:57:56 DEBUG "metric": { 20:57:56 DEBUG "instance": "10.142.0.107:9100" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG [...] 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG 20:57:56 DEBUG "instance": "10.142.0.43:9100" 20:57:56 DEBUG }, 20:57:56 DEBUG "value": [ 20:57:56 DEBUG 1713383876, 20:57:56 DEBUG "0.03133333333333328" 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG ] 20:57:56 DEBUG } 20:57:56 DEBUG } 20:57:57 INFO 20:57:57 INFO [loop_until]: kubectl --namespace=xlou top pods 20:57:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:57:57 INFO [loop_until]: OK (rc = 0) 20:57:57 DEBUG --- stdout --- 20:57:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 1204m 8456Mi am-b59c765b7-bgsxp 1131m 8615Mi am-b59c765b7-ch5q4 1161m 8302Mi ds-cts-0 3174m 7823Mi ds-cts-1 2818m 7604Mi ds-cts-2 3152m 7921Mi ds-idrepo-0 344m 3081Mi ds-idrepo-1 323m 2766Mi ds-idrepo-2 374m 2799Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 965Mi idm-b85f9d978-wnhxb 4m 1003Mi lodemon-db4967764-9zt9t 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 1m 16Mi 20:57:57 DEBUG --- stderr --- 20:57:57 DEBUG 20:57:58 INFO 20:57:58 INFO [loop_until]: kubectl --namespace=xlou top node 20:57:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:57:58 INFO [loop_until]: OK (rc = 0) 20:57:58 DEBUG --- stdout --- 20:57:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 5486Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 700m 2% 6033Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1207m 4% 10925Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 100m 0% 2997Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 3087m 10% 10031Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5024Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1346m 4% 11129Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 72m 0% 2931Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1244m 4% 10507Mi 9% gke-xlou-cdm-ds-0f601751-j488 386m 1% 5264Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 2905m 9% 11022Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 474m 1% 5868Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 3153m 10% 9561Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 70m 0% 3842Mi 6% 20:57:58 DEBUG --- stderr --- 20:57:58 DEBUG 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "ds-idrepo-1" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0.000019612590799030928" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383936" 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "1015.1333333333333" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "instance": "10.142.0.107:9100" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "instance": "10.142.0.43:9100" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0.013833333333333067" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "ds-idrepo-0" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "ds-cts-2" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "ds-idrepo-1" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383936" 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383936" 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0.012" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 DEBUG --- status code --- 20:58:56 ERROR [http_cmd]: ERROR 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 ERROR --- status code --- 20:58:56 DEBUG --- http response --- 20:58:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:58:56 ERROR --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:58:56 DEBUG ----- output ----- 20:58:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "instance": "10.142.0.107:9100" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "instance": "10.142.0.43:9100" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "7550955.5" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "ds-cts-0" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "ds-cts-2" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0.00006666666666666667" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383936" 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "467357696" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "ds-idrepo-1" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "instance": "10.142.0.107:9100" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "instance": "10.142.0.43:9100" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "157638849536" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "84536" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "ds-cts-0" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "ds-cts-2" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "ds-cts-0" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "ds-cts-2" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "29.966666666666665" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "ds-idrepo-0" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "ds-idrepo-2" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0.06666666666666667" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383936" 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "instance": "10.142.0.107:9100" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "instance": "10.142.0.43:9100" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "7542038.733333333" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "1.147345399579448" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "ds-idrepo-0" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "ds-idrepo-2" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0.06666666666666667" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "ds-idrepo-1" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "pod": "ds-idrepo-0" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "pod": "ds-idrepo-2" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0.06666666666666667" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383936" 20:58:56 INFO [http_cmd]: http status code OK 20:58:56 DEBUG --- status code --- 20:58:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:58:56 DEBUG --- http response --- 20:58:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:58:56 DEBUG ----- output ----- 20:58:56 DEBUG { 20:58:56 DEBUG "status": "success", 20:58:56 DEBUG "data": { 20:58:56 DEBUG "resultType": "vector", 20:58:56 DEBUG "result": [ 20:58:56 DEBUG { 20:58:56 DEBUG "metric": { 20:58:56 DEBUG "instance": "10.142.0.107:9100" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG [...] 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG 20:58:56 DEBUG "instance": "10.142.0.43:9100" 20:58:56 DEBUG }, 20:58:56 DEBUG "value": [ 20:58:56 DEBUG 1713383936, 20:58:56 DEBUG "0.010333333333333292" 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG ] 20:58:56 DEBUG } 20:58:56 DEBUG } 20:58:57 INFO 20:58:57 INFO [loop_until]: kubectl --namespace=xlou top pods 20:58:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:58:57 INFO [loop_until]: OK (rc = 0) 20:58:57 DEBUG --- stdout --- 20:58:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 1200m 8446Mi am-b59c765b7-bgsxp 1114m 8611Mi am-b59c765b7-ch5q4 1174m 8281Mi ds-cts-0 2963m 7817Mi ds-cts-1 2751m 7591Mi ds-cts-2 3083m 7900Mi ds-idrepo-0 659m 3131Mi ds-idrepo-1 588m 2831Mi ds-idrepo-2 530m 2755Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 965Mi idm-b85f9d978-wnhxb 4m 1004Mi lodemon-db4967764-9zt9t 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 998m 1054Mi 20:58:57 DEBUG --- stderr --- 20:58:57 DEBUG 20:58:58 INFO 20:58:58 INFO [loop_until]: kubectl --namespace=xlou top node 20:58:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:58:58 INFO [loop_until]: OK (rc = 0) 20:58:58 DEBUG --- stdout --- 20:58:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 5486Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 824m 2% 5978Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1168m 3% 10894Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 2991Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 3037m 10% 10034Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5017Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1262m 4% 11100Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 61m 0% 2897Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1243m 4% 10499Mi 9% gke-xlou-cdm-ds-0f601751-j488 605m 2% 5262Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 2787m 9% 11043Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 520m 1% 5874Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 3062m 10% 9561Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 1080m 6% 4859Mi 8% 20:58:58 DEBUG --- stderr --- 20:58:58 DEBUG 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG "data": { 20:59:56 DEBUG { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG "data": { 20:59:56 DEBUG { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "result": [ 20:59:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:59:56 DEBUG { 20:59:56 DEBUG }, 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "value": [ 20:59:56 DEBUG "pod": "ds-idrepo-0" 20:59:56 DEBUG 20:59:56 DEBUG }, 20:59:56 DEBUG 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:59:56 DEBUG 20:59:56 DEBUG }, 20:59:56 DEBUG 20:59:56 DEBUG "value": [ 20:59:56 DEBUG "pod": "ds-cts-2" 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG }, 20:59:56 DEBUG "988.6666666666666" 20:59:56 DEBUG "value": [ 20:59:56 DEBUG ] 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG } 20:59:56 DEBUG "0" 20:59:56 DEBUG ] 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "data": { 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "instance": "10.142.0.107:9100" 20:59:56 DEBUG "pod": "ds-idrepo-1" 20:59:56 DEBUG }, 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "instance": "10.142.0.43:9100" 20:59:56 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:59:56 DEBUG }, 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "0.014166666666667046" 20:59:56 DEBUG "0.749240866517248" 20:59:56 DEBUG ] 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:59:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:59:56 DEBUG }, 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:59:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:59:56 DEBUG }, 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "0" 20:59:56 DEBUG "0" 20:59:56 DEBUG ] 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "pod": "ds-idrepo-1" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "5628402.052806002" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713383996" 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 ERROR [http_cmd]: ERROR 20:59:56 DEBUG --- status code --- 20:59:56 ERROR --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 ERROR --- http response --- 20:59:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:59:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:59:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:59:56 DEBUG "pod": "ds-cts-0" 20:59:56 DEBUG }, 20:59:56 DEBUG }, 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG "value": [ 20:59:56 DEBUG "value": [ 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG { 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG 20:59:56 DEBUG "pod": "ds-cts-2" 20:59:56 DEBUG "data": { 20:59:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:59:56 DEBUG }, 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG "result": [ 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG { 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "0.00006666666666666667" 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "0.012" 20:59:56 DEBUG ] 20:59:56 DEBUG "instance": "10.142.0.107:9100" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG }, 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG "value": [ 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 DEBUG 20:59:56 DEBUG } 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "instance": "10.142.0.43:9100" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "7458887.666666667" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "0" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "1734250496" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "0" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "pod": "ds-idrepo-1" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "107.04657853527077" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383996" 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "instance": "10.142.0.107:9100" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "instance": "10.142.0.43:9100" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "157787325440" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "pod": "ds-cts-0" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "pod": "ds-cts-2" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "0" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "pod": "am-b59c765b7-ch5q4" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "pod": "am-b59c765b7-bgsxp" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "23840.5" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "pod": "ds-cts-0" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "pod": "ds-cts-2" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "29.966666666666665" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "pod": "ds-idrepo-0" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "pod": "ds-idrepo-2" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "0" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "1.0311319896644293" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "instance": "10.142.0.107:9100" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "instance": "10.142.0.43:9100" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "7449602.699999999" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "pod": "ds-idrepo-0" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "pod": "ds-idrepo-2" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "0" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "pod": "ds-idrepo-0" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "pod": "ds-idrepo-2" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "0" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "pod": "ds-idrepo-1" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "107.04657853527077" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713383996" 20:59:56 INFO [http_cmd]: http status code OK 20:59:56 DEBUG --- status code --- 20:59:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:59:56 DEBUG --- http response --- 20:59:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:59:56 DEBUG ----- output ----- 20:59:56 DEBUG { 20:59:56 DEBUG "status": "success", 20:59:56 DEBUG "data": { 20:59:56 DEBUG "resultType": "vector", 20:59:56 DEBUG "result": [ 20:59:56 DEBUG { 20:59:56 DEBUG "metric": { 20:59:56 DEBUG "instance": "10.142.0.107:9100" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG [...] 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG 20:59:56 DEBUG "instance": "10.142.0.43:9100" 20:59:56 DEBUG }, 20:59:56 DEBUG "value": [ 20:59:56 DEBUG 1713383996, 20:59:56 DEBUG "0.010333333333333231" 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG ] 20:59:56 DEBUG } 20:59:56 DEBUG } 20:59:57 INFO 20:59:57 INFO [loop_until]: kubectl --namespace=xlou top pods 20:59:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:59:57 INFO [loop_until]: OK (rc = 0) 20:59:57 DEBUG --- stdout --- 20:59:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 951m 8444Mi am-b59c765b7-bgsxp 922m 8602Mi am-b59c765b7-ch5q4 1059m 8281Mi ds-cts-0 2835m 7819Mi ds-cts-1 2522m 7596Mi ds-cts-2 2857m 7885Mi ds-idrepo-0 311m 3129Mi ds-idrepo-1 325m 2831Mi ds-idrepo-2 223m 2755Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 8m 966Mi idm-b85f9d978-wnhxb 4m 1004Mi lodemon-db4967764-9zt9t 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 246m 1444Mi 20:59:57 DEBUG --- stderr --- 20:59:57 DEBUG 20:59:58 INFO 20:59:58 INFO [loop_until]: kubectl --namespace=xlou top node 20:59:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:59:58 INFO [loop_until]: OK (rc = 0) 20:59:58 DEBUG --- stdout --- 20:59:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 5486Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 355m 1% 6002Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 999m 3% 10894Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 2999Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2649m 8% 10037Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 5014Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1071m 3% 11101Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 59m 0% 2877Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1071m 3% 10492Mi 9% gke-xlou-cdm-ds-0f601751-j488 349m 1% 5277Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 2442m 8% 11038Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 326m 1% 5906Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 2696m 9% 9556Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 670m 4% 5496Mi 9% 20:59:58 DEBUG --- stderr --- 20:59:58 DEBUG 21:00:21 INFO Finished: True 21:00:21 INFO Waiting for threads to register finish flag 21:00:58 INFO Done. Have a nice day! :) 127.0.0.1 - - [17/Apr/2024 21:00:58] "GET /monitoring/stop HTTP/1.1" 200 - 21:01:01 WARNING File /tmp/lodemon_data-20240417_201944/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6521', ...) 127.0.0.1 - - [17/Apr/2024 21:01:05] "GET /monitoring/process HTTP/1.1" 200 -