==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-69f8bd86c8-8fxnk Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Tue, 16 Apr 2024 17:26:44 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=69f8bd86c8 skaffold.dev/run-id=218da545-cb3b-4de7-a9e1-f573c3253f2d Annotations: Status: Running IP: 10.106.46.152 IPs: IP: 10.106.46.152 Controlled By: ReplicaSet/lodemon-69f8bd86c8 Containers: lodemon: Container ID: containerd://fda84009f9165e12fa07d963636c7050e34365481e6e2c0c1156d9a6c2698a61 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:8ebc4b3f3093af7a1156c744d7231c3add77b4e994da0c73152ecc0b152e1dd6 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Tue, 16 Apr 2024 17:27:01 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h656z (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-h656z: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 16m default-scheduler Successfully assigned xlou/lodemon-69f8bd86c8-8fxnk to gke-xlou-cdm-ds-0f601751-mc5d Normal Pulling 16m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 16m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 16.592184808s (16.592192664s including waiting) Normal Created 16m kubelet Created container lodemon Normal Started 16m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.46.152:8080 Press CTRL+C to quit 18:28:01 INFO 18:28:01 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:01 INFO [loop_until]: OK (rc = 0) 18:28:01 DEBUG --- stdout --- 18:28:01 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:01 DEBUG --- stderr --- 18:28:01 DEBUG 18:28:01 INFO 18:28:01 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:01 INFO [loop_until]: OK (rc = 0) 18:28:01 DEBUG --- stdout --- 18:28:01 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:01 DEBUG --- stderr --- 18:28:01 DEBUG 18:28:01 INFO 18:28:01 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:01 INFO [loop_until]: OK (rc = 0) 18:28:01 DEBUG --- stdout --- 18:28:01 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:01 DEBUG --- stderr --- 18:28:01 DEBUG 18:28:02 INFO 18:28:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:02 INFO [loop_until]: OK (rc = 0) 18:28:02 DEBUG --- stdout --- 18:28:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:02 DEBUG --- stderr --- 18:28:02 DEBUG 18:28:02 INFO 18:28:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:02 INFO [loop_until]: OK (rc = 0) 18:28:02 DEBUG --- stdout --- 18:28:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:02 DEBUG --- stderr --- 18:28:02 DEBUG 18:28:02 INFO 18:28:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:02 INFO [loop_until]: OK (rc = 0) 18:28:02 DEBUG --- stdout --- 18:28:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:02 DEBUG --- stderr --- 18:28:02 DEBUG 18:28:02 INFO 18:28:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:02 INFO [loop_until]: OK (rc = 0) 18:28:02 DEBUG --- stdout --- 18:28:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:02 DEBUG --- stderr --- 18:28:02 DEBUG 18:28:02 INFO 18:28:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:02 INFO [loop_until]: OK (rc = 0) 18:28:02 DEBUG --- stdout --- 18:28:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:02 DEBUG --- stderr --- 18:28:02 DEBUG 18:28:02 INFO 18:28:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:02 INFO [loop_until]: OK (rc = 0) 18:28:02 DEBUG --- stdout --- 18:28:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:02 DEBUG --- stderr --- 18:28:02 DEBUG 18:28:02 INFO 18:28:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:02 INFO [loop_until]: OK (rc = 0) 18:28:02 DEBUG --- stdout --- 18:28:02 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:02 DEBUG --- stderr --- 18:28:02 DEBUG 18:28:02 INFO 18:28:02 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:02 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:03 INFO [loop_until]: OK (rc = 0) 18:28:03 DEBUG --- stdout --- 18:28:03 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:03 DEBUG --- stderr --- 18:28:03 DEBUG 18:28:03 INFO 18:28:03 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:03 INFO [loop_until]: OK (rc = 0) 18:28:03 DEBUG --- stdout --- 18:28:03 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:03 DEBUG --- stderr --- 18:28:03 DEBUG 18:28:03 INFO 18:28:03 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:03 INFO [loop_until]: OK (rc = 0) 18:28:03 DEBUG --- stdout --- 18:28:03 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:03 DEBUG --- stderr --- 18:28:03 DEBUG 18:28:03 INFO 18:28:03 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:03 INFO [loop_until]: OK (rc = 0) 18:28:03 DEBUG --- stdout --- 18:28:03 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:03 DEBUG --- stderr --- 18:28:03 DEBUG 18:28:03 INFO 18:28:03 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:03 INFO [loop_until]: OK (rc = 0) 18:28:03 DEBUG --- stdout --- 18:28:03 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:03 DEBUG --- stderr --- 18:28:03 DEBUG 18:28:03 INFO 18:28:03 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:03 INFO [loop_until]: OK (rc = 0) 18:28:03 DEBUG --- stdout --- 18:28:03 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:03 DEBUG --- stderr --- 18:28:03 DEBUG 18:28:03 INFO 18:28:03 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:03 INFO [loop_until]: OK (rc = 0) 18:28:03 DEBUG --- stdout --- 18:28:03 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:03 DEBUG --- stderr --- 18:28:03 DEBUG 18:28:03 INFO 18:28:03 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:03 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:03 INFO [loop_until]: OK (rc = 0) 18:28:03 DEBUG --- stdout --- 18:28:03 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:03 DEBUG --- stderr --- 18:28:03 DEBUG 18:28:04 INFO 18:28:04 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:04 INFO [loop_until]: OK (rc = 0) 18:28:04 DEBUG --- stdout --- 18:28:04 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:04 DEBUG --- stderr --- 18:28:04 DEBUG 18:28:04 INFO 18:28:04 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:04 INFO [loop_until]: OK (rc = 0) 18:28:04 DEBUG --- stdout --- 18:28:04 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:04 DEBUG --- stderr --- 18:28:04 DEBUG 18:28:04 INFO 18:28:04 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:04 INFO [loop_until]: OK (rc = 0) 18:28:04 DEBUG --- stdout --- 18:28:04 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:04 DEBUG --- stderr --- 18:28:04 DEBUG 18:28:04 INFO 18:28:04 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:04 INFO [loop_until]: OK (rc = 0) 18:28:04 DEBUG --- stdout --- 18:28:04 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:04 DEBUG --- stderr --- 18:28:04 DEBUG 18:28:04 INFO 18:28:04 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:04 INFO [loop_until]: OK (rc = 0) 18:28:04 DEBUG --- stdout --- 18:28:04 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:04 DEBUG --- stderr --- 18:28:04 DEBUG 18:28:04 INFO 18:28:04 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:04 INFO [loop_until]: OK (rc = 0) 18:28:04 DEBUG --- stdout --- 18:28:04 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:04 DEBUG --- stderr --- 18:28:04 DEBUG 18:28:04 INFO 18:28:04 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:04 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:04 INFO [loop_until]: OK (rc = 0) 18:28:04 DEBUG --- stdout --- 18:28:04 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:04 DEBUG --- stderr --- 18:28:04 DEBUG 18:28:05 INFO 18:28:05 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:05 INFO [loop_until]: OK (rc = 0) 18:28:05 DEBUG --- stdout --- 18:28:05 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:05 DEBUG --- stderr --- 18:28:05 DEBUG 18:28:05 INFO 18:28:05 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:28:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:05 INFO [loop_until]: OK (rc = 0) 18:28:05 DEBUG --- stdout --- 18:28:05 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:28:05 DEBUG --- stderr --- 18:28:05 DEBUG 18:28:05 INFO Initializing monitoring instance threads 18:28:05 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 18:28:05 INFO Starting instance threads 18:28:05 INFO 18:28:05 INFO Thread started 18:28:05 INFO [loop_until]: kubectl --namespace=xlou top node 18:28:05 INFO 18:28:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:05 INFO Thread started 18:28:05 INFO [loop_until]: kubectl --namespace=xlou top pods 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:28:05 INFO Thread started 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG --- status code --- 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 INFO Thread started 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG --- status code --- 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 DEBUG --- status code --- 18:28:05 INFO Thread started 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG --- status code --- 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 DEBUG --- status code --- 18:28:05 INFO Thread started 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG { 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG --- http response --- 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG --- http response --- 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:05 DEBUG ----- output ----- 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:05 DEBUG "data": { 18:28:05 DEBUG { 18:28:05 DEBUG { 18:28:05 DEBUG { 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG { 18:28:05 DEBUG --- status code --- 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 INFO Thread started 18:28:05 DEBUG { 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:28:05 DEBUG { 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG { 18:28:05 DEBUG "status": "success", 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 DEBUG "status": "success", 18:28:05 INFO Thread started 18:28:05 DEBUG { 18:28:05 DEBUG "result": [ 18:28:05 DEBUG "data": { 18:28:05 DEBUG "data": { 18:28:05 DEBUG "data": { 18:28:05 DEBUG { 18:28:05 INFO [loop_until]: OK (rc = 0) 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG "data": { 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG "data": { 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 DEBUG "status": "success", 18:28:05 INFO Thread started 18:28:05 DEBUG { 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG --- stdout --- 18:28:05 DEBUG { 18:28:05 DEBUG { 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG "data": { 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG "data": { 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 DEBUG "metric": { 18:28:05 INFO Thread started 18:28:05 DEBUG "result": [ 18:28:05 INFO [loop_until]: OK (rc = 0) 18:28:05 DEBUG "result": [ 18:28:05 DEBUG "result": [ 18:28:05 DEBUG "data": { 18:28:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 6m 2930Mi am-555f4db595-fz8sn 7m 3180Mi am-555f4db595-kzxsp 7m 1120Mi ds-cts-0 5m 8289Mi ds-cts-1 5m 8306Mi ds-cts-2 4m 8446Mi ds-idrepo-0 8m 5029Mi ds-idrepo-1 9m 2778Mi ds-idrepo-2 7m 2858Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1041Mi idm-b85f9d978-rl76f 4m 1227Mi lodemon-69f8bd86c8-8fxnk 2m 55Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 1496Mi 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG "data": { 18:28:05 DEBUG "data": { 18:28:05 DEBUG { 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "result": [ 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG "result": [ 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG { 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "pod": "ds-cts-1" 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG { 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 DEBUG --- stdout --- 18:28:05 INFO Thread started 18:28:05 DEBUG { 18:28:05 DEBUG { 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG --- stderr --- 18:28:05 DEBUG "data": { 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "result": [ 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG { 18:28:05 DEBUG { 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG "result": [ 18:28:05 DEBUG }, 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG "metric": { 18:28:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5084Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 10814Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 60m 0% 5414Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 78m 0% 2634Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 60m 0% 6121Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 13007Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2991Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 57m 0% 4677Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 64m 0% 5444Mi 4% gke-xlou-cdm-ds-0f601751-j488 56m 0% 10329Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 99m 0% 3227Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 85m 0% 3006Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 61m 0% 7329Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 67m 0% 5400Mi 9% 18:28:05 DEBUG "metric": { 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 INFO Thread started 18:28:05 DEBUG "metric": { 18:28:05 DEBUG "result": [ 18:28:05 DEBUG 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "result": [ 18:28:05 DEBUG "result": [ 18:28:05 DEBUG { 18:28:05 DEBUG "metric": { 18:28:05 DEBUG "metric": { 18:28:05 DEBUG "metric": { 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG "data": { 18:28:05 DEBUG { 18:28:05 DEBUG "value": [ 18:28:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:28:05 DEBUG --- stderr --- 18:28:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:28:05 DEBUG --- status code --- 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713288485" 18:28:05 DEBUG "pod": "ds-cts-1" 18:28:05 INFO Thread started 18:28:05 DEBUG { 18:28:05 DEBUG "result": [ 18:28:05 DEBUG { 18:28:05 DEBUG { 18:28:05 DEBUG "data": { 18:28:05 DEBUG "pod": "ds-idrepo-2" 18:28:05 DEBUG "pod": "ds-cts-1" 18:28:05 DEBUG "pod": "ds-cts-1" 18:28:05 DEBUG { 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:05 DEBUG "metric": { 18:28:05 DEBUG 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG }, 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:05 DEBUG 18:28:05 DEBUG }, 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG }, 18:28:05 DEBUG "metric": { 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288485" 18:28:05 DEBUG { 18:28:05 INFO Thread started 18:28:05 DEBUG "metric": { 18:28:05 DEBUG "metric": { 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG }, 18:28:05 DEBUG "value": [ 18:28:05 DEBUG }, 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG "result": [ 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG "instance": "10.142.0.107:9100" 18:28:05 DEBUG 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG "value": [ 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG "value": [ 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:28:05 DEBUG "metric": { 18:28:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288485" 18:28:05 DEBUG "instance": "10.142.0.107:9100" 18:28:05 INFO Thread started 18:28:05 DEBUG "result": [ 18:28:05 DEBUG }, 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "data": { 18:28:05 DEBUG { 18:28:05 DEBUG { 18:28:05 DEBUG }, 18:28:05 DEBUG 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG 18:28:05 DEBUG { 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:28:05 DEBUG }, 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG "instance": "10.142.0.107:9100" 18:28:05 DEBUG }, 18:28:05 DEBUG }, 18:28:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288485" 18:28:05 DEBUG { 18:28:05 INFO Thread started 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "metric": { 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG "value": [ 18:28:05 DEBUG [...] 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG "value": [ 18:28:05 DEBUG --- status code --- 18:28:05 ERROR [http_cmd]: ERROR 18:28:05 DEBUG }, 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "metric": { 18:28:05 INFO All threads has been started 127.0.0.1 - - [16/Apr/2024 18:28:05] "GET /monitoring/start HTTP/1.1" 200 - 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG "result": [ 18:28:05 DEBUG "pod": "ds-idrepo-2" 18:28:05 DEBUG "data": { 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:28:05 DEBUG 18:28:05 DEBUG "data": { 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG { 18:28:05 DEBUG 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 ERROR --- status code --- 18:28:05 DEBUG "value": [ 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG "pod": "ds-idrepo-2" 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG { 18:28:05 DEBUG }, 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG [...] 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG [...] 18:28:05 DEBUG [...] 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG 18:28:05 DEBUG --- http response --- 18:28:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG }, 18:28:05 DEBUG [...] 18:28:05 DEBUG [...] 18:28:05 DEBUG [...] 18:28:05 DEBUG "metric": { 18:28:05 DEBUG "value": [ 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG "result": [ 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG { 18:28:05 DEBUG { 18:28:05 DEBUG { 18:28:05 DEBUG 18:28:05 DEBUG "result": [ 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG "data": { 18:28:05 DEBUG 18:28:05 ERROR --- http response --- 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG "value": [ 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG "pod": "ds-idrepo-2" 18:28:05 DEBUG 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG { 18:28:05 DEBUG [...] 18:28:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG 18:28:05 DEBUG { 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG [...] 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG 18:28:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:28:05 DEBUG [...] 18:28:05 DEBUG [...] 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG }, 18:28:05 DEBUG 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG "metric": { 18:28:05 DEBUG 18:28:05 DEBUG }, 18:28:05 DEBUG "data": { 18:28:05 DEBUG "data": { 18:28:05 DEBUG "data": { 18:28:05 DEBUG 18:28:05 DEBUG "metric": { 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG "result": [ 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:05 DEBUG 18:28:05 DEBUG { 18:28:05 DEBUG [...] 18:28:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG "value": [ 18:28:05 DEBUG 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:28:05 DEBUG 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:28:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:28:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:28:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:28:05 DEBUG { 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG "pod": "ds-idrepo-0" 18:28:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:28:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:28:05 DEBUG 18:28:05 DEBUG [...] 18:28:05 DEBUG }, 18:28:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:28:05 DEBUG 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG "result": [ 18:28:05 DEBUG "result": [ 18:28:05 DEBUG "result": [ 18:28:05 DEBUG }, 18:28:05 DEBUG }, 18:28:05 DEBUG }, 18:28:05 DEBUG }, 18:28:05 DEBUG "metric": { 18:28:05 DEBUG { 18:28:05 DEBUG 18:28:05 DEBUG "data": { 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG [...] 18:28:05 DEBUG }, 18:28:05 DEBUG }, 18:28:05 DEBUG }, 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "instance": "10.142.0.43:9100" 18:28:05 DEBUG "0" 18:28:05 DEBUG { 18:28:05 DEBUG { 18:28:05 DEBUG { 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "pod": "ds-idrepo-2" 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG 18:28:05 DEBUG "instance": "10.142.0.43:9100" 18:28:05 DEBUG "instance": "10.142.0.43:9100" 18:28:05 DEBUG 18:28:05 DEBUG "value": [ 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG "value": [ 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG }, 18:28:05 DEBUG ] 18:28:05 DEBUG "metric": { 18:28:05 DEBUG "metric": { 18:28:05 DEBUG "metric": { 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG }, 18:28:05 DEBUG "data": { 18:28:05 DEBUG }, 18:28:05 DEBUG "result": [ 18:28:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:28:05 DEBUG }, 18:28:05 DEBUG }, 18:28:05 DEBUG 18:28:05 DEBUG "value": [ 18:28:05 INFO [http_cmd]: http status code OK 18:28:05 DEBUG "0" 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG [...] 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG "value": [ 18:28:05 DEBUG } 18:28:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:28:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:28:05 DEBUG "pod": "ds-idrepo-2" 18:28:05 DEBUG "1668931584" 18:28:05 DEBUG 18:28:05 DEBUG "0.03169788465860182" 18:28:05 DEBUG "0" 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG "value": [ 18:28:05 DEBUG { 18:28:05 DEBUG }, 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "value": [ 18:28:05 DEBUG 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG --- status code --- 18:28:05 DEBUG ] 18:28:05 DEBUG "0.00002232833854766127" 18:28:05 DEBUG 18:28:05 DEBUG "pod": "ds-idrepo-0" 18:28:05 DEBUG 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG ] 18:28:05 DEBUG }, 18:28:05 DEBUG }, 18:28:05 DEBUG }, 18:28:05 DEBUG ] 18:28:05 DEBUG 18:28:05 DEBUG ] 18:28:05 DEBUG ] 18:28:05 DEBUG 18:28:05 DEBUG "result": [ 18:28:05 DEBUG { 18:28:05 DEBUG "metric": { 18:28:05 DEBUG "value": [ 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG "pod": "ds-idrepo-0" 18:28:05 DEBUG "0" 18:28:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:28:05 DEBUG } 18:28:05 DEBUG ] 18:28:05 DEBUG 18:28:05 DEBUG }, 18:28:05 DEBUG [...] 18:28:05 DEBUG "0.2526666666666667" 18:28:05 DEBUG } 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "value": [ 18:28:05 DEBUG } 18:28:05 DEBUG [...] 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG "metric": { 18:28:05 DEBUG "pod": "ds-idrepo-2" 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG "15809.633333333331" 18:28:05 DEBUG "10951.333333333334" 18:28:05 DEBUG }, 18:28:05 DEBUG ] 18:28:05 DEBUG --- http response --- 18:28:05 DEBUG ] 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG "value": [ 18:28:05 DEBUG 18:28:05 DEBUG ] 18:28:05 DEBUG } 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG ] 18:28:05 DEBUG 18:28:05 DEBUG ] 18:28:05 DEBUG ] 18:28:05 DEBUG 18:28:05 DEBUG "0" 18:28:05 DEBUG "instance": "10.142.0.107:9100" 18:28:05 DEBUG }, 18:28:05 DEBUG "0" 18:28:05 DEBUG ] 18:28:05 DEBUG ] 18:28:05 DEBUG "value": [ 18:28:05 DEBUG } 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:28:05 DEBUG ] 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG [...] 18:28:05 DEBUG ] 18:28:05 DEBUG }, 18:28:05 DEBUG "value": [ 18:28:05 DEBUG ] 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG ] 18:28:05 DEBUG "pod": "ds-idrepo-0" 18:28:05 DEBUG ----- output ----- 18:28:05 DEBUG } 18:28:05 DEBUG "0" 18:28:05 DEBUG 18:28:05 DEBUG ] 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG "value": [ 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG ] 18:28:05 DEBUG ] 18:28:05 DEBUG "0" 18:28:05 DEBUG } 18:28:05 DEBUG }, 18:28:05 DEBUG { 18:28:05 DEBUG } 18:28:05 DEBUG ] 18:28:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:28:05 DEBUG } 18:28:05 DEBUG [...] 18:28:05 DEBUG [...] 18:28:05 DEBUG [...] 18:28:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:28:05 DEBUG 18:28:05 DEBUG ] 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG ] 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG ] 18:28:05 DEBUG } 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "status": "success", 18:28:05 DEBUG } 18:28:05 DEBUG }, 18:28:05 DEBUG } 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG }, 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG "data": { 18:28:05 DEBUG ] 18:28:05 DEBUG "value": [ 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "pod": "ds-idrepo-0" 18:28:05 DEBUG } 18:28:05 DEBUG 18:28:05 DEBUG [...] 18:28:05 DEBUG } 18:28:05 DEBUG ] 18:28:05 DEBUG "0" 18:28:05 DEBUG "resultType": "vector", 18:28:05 DEBUG } 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG }, 18:28:05 DEBUG [...] 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG ] 18:28:05 DEBUG "result": [ 18:28:05 DEBUG } 18:28:05 DEBUG "0" 18:28:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:28:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:28:05 DEBUG "pod": "ds-idrepo-0" 18:28:05 DEBUG "0" 18:28:05 DEBUG "value": [ 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG { 18:28:05 DEBUG ] 18:28:05 DEBUG }, 18:28:05 DEBUG }, 18:28:05 DEBUG }, 18:28:05 DEBUG ] 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG 18:28:05 DEBUG 18:28:05 DEBUG ] 18:28:05 DEBUG "metric": { 18:28:05 DEBUG } 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "value": [ 18:28:05 DEBUG } 18:28:05 DEBUG "0" 18:28:05 DEBUG 18:28:05 DEBUG "pod": "ds-cts-2" 18:28:05 DEBUG } 18:28:05 DEBUG "instance": "10.142.0.107:9100" 18:28:05 DEBUG ] 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG ] 18:28:05 DEBUG ] 18:28:05 DEBUG "instance": "10.142.0.43:9100" 18:28:05 DEBUG }, 18:28:05 DEBUG } 18:28:05 DEBUG }, 18:28:05 DEBUG } 18:28:05 DEBUG "2" 18:28:05 DEBUG "0.005" 18:28:05 DEBUG "0" 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG }, 18:28:05 DEBUG "value": [ 18:28:05 DEBUG "value": [ 18:28:05 DEBUG } 18:28:05 DEBUG ] 18:28:05 DEBUG ] 18:28:05 DEBUG ] 18:28:05 DEBUG } 18:28:05 DEBUG ] 18:28:05 DEBUG "value": [ 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG "0" 18:28:05 DEBUG 18:28:05 DEBUG ] 18:28:05 DEBUG ] 18:28:05 DEBUG ] 18:28:05 DEBUG } 18:28:05 DEBUG "18591788032" 18:28:05 DEBUG ] 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG ] 18:28:05 DEBUG } 18:28:05 DEBUG [...] 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG ] 18:28:05 DEBUG 18:28:05 DEBUG ] 18:28:05 DEBUG } 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG } 18:28:05 DEBUG 18:28:05 DEBUG } 18:28:05 DEBUG "instance": "10.142.0.43:9100" 18:28:05 DEBUG }, 18:28:05 DEBUG "value": [ 18:28:05 DEBUG 1713288485, 18:28:05 DEBUG "0.2622666666666665" 18:28:05 DEBUG ] 18:28:05 DEBUG } 18:28:05 DEBUG ] 18:28:05 DEBUG } 18:28:05 DEBUG } 18:29:05 INFO 18:29:05 INFO [loop_until]: kubectl --namespace=xlou top pods 18:29:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 DEBUG --- status code --- 18:29:05 INFO [loop_until]: OK (rc = 0) 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG --- stdout --- 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 13m 2979Mi am-555f4db595-fz8sn 4m 3183Mi am-555f4db595-kzxsp 18m 1121Mi ds-cts-0 5m 8290Mi ds-cts-1 4m 8306Mi ds-cts-2 4m 8447Mi ds-idrepo-0 8m 5029Mi ds-idrepo-1 8m 2779Mi ds-idrepo-2 32m 2857Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1042Mi idm-b85f9d978-rl76f 5m 1226Mi lodemon-69f8bd86c8-8fxnk 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 5Mi overseer-0-5584f6c4db-bcn5q 1m 1496Mi 18:29:05 DEBUG --- status code --- 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG --- stderr --- 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG { 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG ----- output ----- 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG --- status code --- 18:29:05 INFO [loop_until]: kubectl --namespace=xlou top node 18:29:05 DEBUG "data": { 18:29:05 DEBUG { 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG { 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "data": { 18:29:05 DEBUG --- http response --- 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 DEBUG "data": { 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:29:05 DEBUG { 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG "resultType": "vector", 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "result": [ 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG "result": [ 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG { 18:29:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:29:05 DEBUG { 18:29:05 DEBUG { 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG { 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG }, 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:29:05 DEBUG "metric": { 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG "data": { 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "pod": "ds-idrepo-2" 18:29:05 DEBUG "data": { 18:29:05 DEBUG "instance": "10.142.0.107:9100" 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:29:05 DEBUG 18:29:05 DEBUG }, 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG }, 18:29:05 DEBUG { 18:29:05 DEBUG "result": [ 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG { 18:29:05 DEBUG { 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG { 18:29:05 DEBUG 18:29:05 DEBUG "data": { 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG [...] 18:29:05 DEBUG 18:29:05 DEBUG "metric": { 18:29:05 DEBUG 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "pod": "ds-idrepo-2" 18:29:05 DEBUG "data": { 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG "instance": "10.142.0.107:9100" 18:29:05 DEBUG 18:29:05 DEBUG "result": [ 18:29:05 DEBUG }, 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG 18:29:05 DEBUG [...] 18:29:05 DEBUG }, 18:29:05 DEBUG [...] 18:29:05 DEBUG { 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "result": [ 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 18:29:05 DEBUG "metric": { 18:29:05 DEBUG 18:29:05 DEBUG { 18:29:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:29:05 DEBUG 18:29:05 DEBUG "metric": { 18:29:05 DEBUG }, 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG }, 18:29:05 DEBUG 18:29:05 DEBUG "pod": "ds-idrepo-2" 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "pod": "ds-idrepo-0" 18:29:05 DEBUG 18:29:05 DEBUG "instance": "10.142.0.43:9100" 18:29:05 DEBUG "value": [ 18:29:05 DEBUG [...] 18:29:05 DEBUG }, 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG }, 18:29:05 DEBUG [...] 18:29:05 DEBUG }, 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "1666461696" 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG ] 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG } 18:29:05 DEBUG "0" 18:29:05 DEBUG 18:29:05 DEBUG "8994.8" 18:29:05 DEBUG [...] 18:29:05 DEBUG "pod": "ds-idrepo-0" 18:29:05 DEBUG 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG "instance": "10.142.0.43:9100" 18:29:05 DEBUG ] 18:29:05 DEBUG 18:29:05 DEBUG }, 18:29:05 DEBUG [...] 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG }, 18:29:05 DEBUG } 18:29:05 DEBUG 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG "value": [ 18:29:05 DEBUG ] 18:29:05 DEBUG 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG 18:29:05 DEBUG } 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG } 18:29:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:29:05 DEBUG "0" 18:29:05 DEBUG 18:29:05 DEBUG } 18:29:05 DEBUG "19828950016" 18:29:05 DEBUG } 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 DEBUG }, 18:29:05 DEBUG ] 18:29:05 DEBUG "pod": "ds-cts-2" 18:29:05 DEBUG ] 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713288545" 18:29:05 DEBUG "value": [ 18:29:05 DEBUG } 18:29:05 DEBUG }, 18:29:05 DEBUG } 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG ] 18:29:05 DEBUG "value": [ 18:29:05 DEBUG ] 18:29:05 DEBUG "0" 18:29:05 DEBUG } 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG "0" 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 ERROR [http_cmd]: ERROR 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 ERROR --- status code --- 18:29:05 DEBUG --- status code --- 18:29:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 ERROR --- http response --- 18:29:05 DEBUG --- http response --- 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:29:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG { 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG "data": { 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "result": [ 18:29:05 DEBUG { 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "pod": "ds-idrepo-2" 18:29:05 DEBUG }, 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG [...] 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG "pod": "ds-idrepo-0" 18:29:05 DEBUG }, 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG "0" 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713288545" 18:29:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288545" 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 DEBUG --- status code --- 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG --- status code --- 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG --- http response --- 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:05 INFO [http_cmd]: http status code OK 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG --- status code --- 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:05 DEBUG { 18:29:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG { 18:29:05 DEBUG { 18:29:05 DEBUG { 18:29:05 DEBUG { 18:29:05 DEBUG { 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG --- http response --- 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG { 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:05 DEBUG { 18:29:05 DEBUG { 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:05 DEBUG { 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:29:05 DEBUG "data": { 18:29:05 DEBUG { 18:29:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:29:05 DEBUG { 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG "data": { 18:29:05 DEBUG "data": { 18:29:05 DEBUG "data": { 18:29:05 DEBUG "data": { 18:29:05 DEBUG "data": { 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG ----- output ----- 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG "data": { 18:29:05 DEBUG { 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG "data": { 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG { 18:29:05 DEBUG "data": { 18:29:05 DEBUG { 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "data": { 18:29:05 DEBUG { 18:29:05 DEBUG "data": { 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "data": { 18:29:05 DEBUG "data": { 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG { 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "status": "success", 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "result": [ 18:29:05 DEBUG { 18:29:05 DEBUG { 18:29:05 DEBUG { 18:29:05 DEBUG { 18:29:05 DEBUG { 18:29:05 DEBUG "data": { 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "data": { 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "data": { 18:29:05 DEBUG "result": [ 18:29:05 DEBUG { 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "result": [ 18:29:05 DEBUG { 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG { 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:29:05 DEBUG { 18:29:05 DEBUG "resultType": "vector", 18:29:05 DEBUG { 18:29:05 DEBUG "metric": { 18:29:05 DEBUG { 18:29:05 DEBUG { 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:29:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:29:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:29:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:29:05 DEBUG "instance": "10.142.0.107:9100" 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "result": [ 18:29:05 DEBUG }, 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "result": [ 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "instance": "10.142.0.107:9100" 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "metric": { 18:29:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:29:05 DEBUG }, 18:29:05 DEBUG }, 18:29:05 DEBUG }, 18:29:05 DEBUG }, 18:29:05 DEBUG }, 18:29:05 DEBUG { 18:29:05 DEBUG "pod": "ds-idrepo-2" 18:29:05 DEBUG { 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "pod": "ds-idrepo-2" 18:29:05 DEBUG { 18:29:05 DEBUG "pod": "ds-idrepo-2" 18:29:05 DEBUG }, 18:29:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:29:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:29:05 DEBUG }, 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "metric": { 18:29:05 DEBUG }, 18:29:05 DEBUG "metric": { 18:29:05 DEBUG 18:29:05 DEBUG }, 18:29:05 DEBUG "metric": { 18:29:05 DEBUG }, 18:29:05 DEBUG "value": [ 18:29:05 DEBUG }, 18:29:05 DEBUG }, 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:29:05 DEBUG 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "instance": "10.142.0.107:9100" 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG }, 18:29:05 DEBUG 18:29:05 DEBUG }, 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG }, 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 18:29:05 DEBUG "value": [ 18:29:05 DEBUG [...] 18:29:05 DEBUG 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG [...] 18:29:05 DEBUG [...] 18:29:05 DEBUG [...] 18:29:05 DEBUG [...] 18:29:05 DEBUG [...] 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG [...] 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG [...] 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG [...] 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG [...] 18:29:05 DEBUG 18:29:05 DEBUG [...] 18:29:05 DEBUG 18:29:05 DEBUG [...] 18:29:05 DEBUG [...] 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG [...] 18:29:05 DEBUG 18:29:05 DEBUG [...] 18:29:05 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 18:29:05 DEBUG 18:29:05 DEBUG [...] 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:29:05 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 18:29:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:29:05 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 18:29:05 DEBUG "instance": "10.142.0.43:9100" 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG }, 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG "instance": "10.142.0.43:9100" 18:29:05 DEBUG 18:29:05 DEBUG 18:29:05 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 18:29:05 DEBUG }, 18:29:05 DEBUG }, 18:29:05 DEBUG }, 18:29:05 DEBUG }, 18:29:05 DEBUG }, 18:29:05 DEBUG 18:29:05 DEBUG "pod": "ds-idrepo-0" 18:29:05 DEBUG 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "pod": "ds-idrepo-0" 18:29:05 DEBUG 18:29:05 DEBUG "pod": "ds-idrepo-0" 18:29:05 DEBUG }, 18:29:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:29:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:29:05 DEBUG }, 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 18:29:05 DEBUG }, 18:29:05 DEBUG 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG }, 18:29:05 DEBUG 18:29:05 DEBUG }, 18:29:05 DEBUG "value": [ 18:29:05 DEBUG }, 18:29:05 DEBUG }, 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:29:05 DEBUG "27670.427609746843" 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "instance": "10.142.0.43:9100" 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "value": [ 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG "0" 18:29:05 DEBUG "0.0014256749646975944" 18:29:05 DEBUG "0.027067192000859167" 18:29:05 DEBUG "0" 18:29:05 DEBUG "16134.23333333333" 18:29:05 DEBUG }, 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG }, 18:29:05 DEBUG ] 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG }, 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG "0.0006666666666666524" 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG "0" 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "0" 18:29:05 DEBUG "value": [ 18:29:05 DEBUG } 18:29:05 DEBUG "0" 18:29:05 DEBUG "value": [ 18:29:05 DEBUG "0" 18:29:05 DEBUG ] 18:29:05 DEBUG "0" 18:29:05 DEBUG "0" 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG ] 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG 1713288545, 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG "0.005" 18:29:05 DEBUG } 18:29:05 DEBUG "2" 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG "0.0005666666666665302" 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG ] 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 DEBUG } 18:29:05 INFO [loop_until]: OK (rc = 0) 18:29:05 DEBUG --- stdout --- 18:29:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5086Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 67m 0% 10800Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 51m 0% 5418Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 2653Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 92m 0% 6120Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 149m 0% 12991Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2974Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 69m 0% 4732Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5439Mi 4% gke-xlou-cdm-ds-0f601751-j488 57m 0% 10338Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 85m 0% 3229Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 95m 0% 2995Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 62m 0% 7318Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 66m 0% 5404Mi 9% 18:29:05 DEBUG --- stderr --- 18:29:05 DEBUG 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288605" 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288605" 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "instance": "10.142.0.107:9100" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "instance": "10.142.0.43:9100" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "9098.5" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "pod": "ds-idrepo-2" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "pod": "ds-cts-2" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "0" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "instance": "10.142.0.107:9100" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "instance": "10.142.0.43:9100" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "19830354944" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "pod": "ds-idrepo-2" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "pod": "ds-idrepo-0" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "0" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 INFO 18:30:05 INFO [loop_until]: kubectl --namespace=xlou top pods 18:30:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "pod": "ds-idrepo-2" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "pod": "ds-idrepo-0" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "0" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713288605" 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "1667649536" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "0" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 ERROR [http_cmd]: ERROR 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:30:05 ERROR --- status code --- 18:30:05 DEBUG ----- output ----- 18:30:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:30:05 DEBUG { 18:30:05 ERROR --- http response --- 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "pod": "ds-idrepo-2" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "pod": "ds-idrepo-0" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "0" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:30:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "0" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "pod": "ds-idrepo-2" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "pod": "ds-idrepo-0" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "0" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "2" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "0" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 INFO [loop_until]: OK (rc = 0) 18:30:05 DEBUG --- stdout --- 18:30:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 6m 3028Mi am-555f4db595-fz8sn 5m 3183Mi am-555f4db595-kzxsp 8m 1120Mi ds-cts-0 5m 8289Mi ds-cts-1 4m 8306Mi ds-cts-2 4m 8446Mi ds-idrepo-0 8m 5029Mi ds-idrepo-1 8m 2779Mi ds-idrepo-2 8m 2858Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1042Mi idm-b85f9d978-rl76f 4m 1226Mi lodemon-69f8bd86c8-8fxnk 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 2m 1497Mi 18:30:05 DEBUG --- stderr --- 18:30:05 DEBUG 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "0.03680204514984496" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "pod": "ds-idrepo-2" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "pod": "ds-idrepo-0" 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "0" 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713288605" 18:30:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288605" 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG --- status code --- 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:30:05 DEBUG --- http response --- 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG ----- output ----- 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 18:30:05 DEBUG --- status code --- 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG { 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG { 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG { 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG "data": { 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:05 DEBUG "data": { 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:05 DEBUG "data": { 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG { 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG { 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "data": { 18:30:05 INFO [http_cmd]: http status code OK 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "data": { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG { 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG --- status code --- 18:30:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "data": { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "data": { 18:30:05 DEBUG "data": { 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG }, 18:30:05 DEBUG "result": [ 18:30:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:30:05 DEBUG }, 18:30:05 DEBUG "result": [ 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG "result": [ 18:30:05 DEBUG "value": [ 18:30:05 DEBUG { 18:30:05 DEBUG --- http response --- 18:30:05 DEBUG "value": [ 18:30:05 DEBUG { 18:30:05 DEBUG "result": [ 18:30:05 DEBUG }, 18:30:05 DEBUG "result": [ 18:30:05 DEBUG "result": [ 18:30:05 DEBUG { 18:30:05 DEBUG 18:30:05 DEBUG "metric": { 18:30:05 DEBUG 18:30:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:30:05 DEBUG "metric": { 18:30:05 DEBUG { 18:30:05 DEBUG "value": [ 18:30:05 DEBUG { 18:30:05 DEBUG { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG 18:30:05 DEBUG "pod": "ds-idrepo-2" 18:30:05 DEBUG 18:30:05 DEBUG ----- output ----- 18:30:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:30:05 DEBUG "metric": { 18:30:05 DEBUG 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "metric": { 18:30:05 DEBUG "instance": "10.142.0.107:9100" 18:30:05 DEBUG 18:30:05 DEBUG }, 18:30:05 DEBUG 18:30:05 DEBUG { 18:30:05 DEBUG }, 18:30:05 DEBUG "instance": "10.142.0.107:9100" 18:30:05 DEBUG 18:30:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:30:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:30:05 DEBUG }, 18:30:05 DEBUG [...] 18:30:05 DEBUG "value": [ 18:30:05 DEBUG [...] 18:30:05 DEBUG "status": "success", 18:30:05 DEBUG "value": [ 18:30:05 DEBUG }, 18:30:05 DEBUG 18:30:05 DEBUG }, 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "data": { 18:30:05 DEBUG 18:30:05 DEBUG "value": [ 18:30:05 DEBUG [...] 18:30:05 DEBUG "value": [ 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "resultType": "vector", 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "result": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "pod": "ds-cts-2" 18:30:05 DEBUG [...] 18:30:05 DEBUG "pod": "ds-cts-2" 18:30:05 DEBUG { 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG }, 18:30:05 DEBUG 18:30:05 DEBUG }, 18:30:05 DEBUG "metric": { 18:30:05 DEBUG 18:30:05 DEBUG [...] 18:30:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:30:05 DEBUG [...] 18:30:05 DEBUG [...] 18:30:05 DEBUG 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG "value": [ 18:30:05 DEBUG "instance": "10.142.0.107:9100" 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG }, 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG }, 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "0" 18:30:05 DEBUG "pod": "ds-idrepo-0" 18:30:05 DEBUG "0" 18:30:05 DEBUG "value": [ 18:30:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:30:05 DEBUG 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG "instance": "10.142.0.43:9100" 18:30:05 DEBUG ] 18:30:05 DEBUG }, 18:30:05 DEBUG ] 18:30:05 DEBUG 18:30:05 DEBUG }, 18:30:05 DEBUG "instance": "10.142.0.43:9100" 18:30:05 DEBUG "495.04471839497216" 18:30:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:30:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:30:05 DEBUG }, 18:30:05 DEBUG } 18:30:05 DEBUG "value": [ 18:30:05 DEBUG } 18:30:05 DEBUG 18:30:05 DEBUG "value": [ 18:30:05 DEBUG }, 18:30:05 DEBUG ] 18:30:05 DEBUG }, 18:30:05 DEBUG }, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG ] 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG ] 18:30:05 DEBUG 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "value": [ 18:30:05 DEBUG } 18:30:05 DEBUG "value": [ 18:30:05 DEBUG "value": [ 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG } 18:30:05 DEBUG "0" 18:30:05 DEBUG } 18:30:05 DEBUG [...] 18:30:05 DEBUG "0" 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG ] 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG "16151.699999999999" 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG 18:30:05 DEBUG 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 DEBUG "0.00033333333333333365" 18:30:05 DEBUG 18:30:05 DEBUG ] 18:30:05 DEBUG "0" 18:30:05 DEBUG } 18:30:05 DEBUG "0.005" 18:30:05 DEBUG ] 18:30:05 DEBUG ] 18:30:05 DEBUG "instance": "10.142.0.43:9100" 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 DEBUG }, 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG "value": [ 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG 1713288605, 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 DEBUG "0.00030000000000001136" 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG ] 18:30:05 DEBUG } 18:30:05 DEBUG } 18:30:05 INFO 18:30:05 INFO [loop_until]: kubectl --namespace=xlou top node 18:30:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:30:05 INFO [loop_until]: OK (rc = 0) 18:30:05 DEBUG --- stdout --- 18:30:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5083Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 67m 0% 10812Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 5420Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 2657Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 64m 0% 6134Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 127m 0% 12996Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2984Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 56m 0% 4786Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 60m 0% 5451Mi 4% gke-xlou-cdm-ds-0f601751-j488 55m 0% 10336Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 83m 0% 3228Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 87m 0% 2979Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 57m 0% 7325Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 71m 0% 5406Mi 9% 18:30:05 DEBUG --- stderr --- 18:30:05 DEBUG 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288665" 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "instance": "10.142.0.107:9100" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "instance": "10.142.0.43:9100" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "10531.433333333332" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "pod": "ds-idrepo-2" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "pod": "ds-idrepo-0" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "pod": "ds-idrepo-2" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "pod": "ds-idrepo-0" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "pod": "ds-idrepo-2" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "pod": "ds-cts-2" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288665" 18:31:05 INFO 18:31:05 INFO [loop_until]: kubectl --namespace=xlou top pods 18:31:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "instance": "10.142.0.107:9100" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "instance": "10.142.0.43:9100" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "19830963200" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "pod": "ds-idrepo-2" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "pod": "ds-idrepo-0" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0" 18:31:05 DEBUG ] 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713288665" 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713288665" 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 DEBUG ] 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG } 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG } 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "2" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 ERROR [http_cmd]: ERROR 18:31:05 DEBUG --- status code --- 18:31:05 ERROR --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 ERROR --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "pod": "ds-idrepo-2" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "pod": "ds-idrepo-0" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:31:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "1669177344" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [loop_until]: OK (rc = 0) 18:31:05 DEBUG --- stdout --- 18:31:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 7m 3077Mi am-555f4db595-fz8sn 4m 3182Mi am-555f4db595-kzxsp 4m 1120Mi ds-cts-0 22m 8290Mi ds-cts-1 5m 8305Mi ds-cts-2 4m 8446Mi ds-idrepo-0 8m 5029Mi ds-idrepo-1 8m 2778Mi ds-idrepo-2 8m 2858Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1042Mi idm-b85f9d978-rl76f 4m 1226Mi lodemon-69f8bd86c8-8fxnk 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 5Mi overseer-0-5584f6c4db-bcn5q 1m 1497Mi 18:31:05 DEBUG --- stderr --- 18:31:05 DEBUG 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288665" 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713288665" 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "instance": "10.142.0.107:9100" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "instance": "10.142.0.43:9100" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "16298.733333333332" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0.026247053971841648" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288665" 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "pod": "ds-idrepo-2" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "pod": "ds-idrepo-0" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "instance": "10.142.0.107:9100" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "instance": "10.142.0.43:9100" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "pod": "ds-idrepo-2" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "pod": "ds-idrepo-0" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288665" 18:31:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288665" 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG "data": { 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "pod": "ds-cts-2" 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0.0035891553703909476" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG --- http response --- 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG { 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:31:05 DEBUG "data": { 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG { 18:31:05 DEBUG "result": [ 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG { 18:31:05 DEBUG "status": "success", 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG "data": { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "data": { 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG "result": [ 18:31:05 DEBUG }, 18:31:05 DEBUG "result": [ 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG { 18:31:05 DEBUG "value": [ 18:31:05 DEBUG { 18:31:05 DEBUG "metric": { 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:31:05 DEBUG 18:31:05 DEBUG "metric": { 18:31:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG 18:31:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:31:05 DEBUG }, 18:31:05 DEBUG { 18:31:05 DEBUG 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG [...] 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG "data": { 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "result": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG [...] 18:31:05 DEBUG { 18:31:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:31:05 DEBUG [...] 18:31:05 DEBUG 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG "metric": { 18:31:05 DEBUG }, 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG }, 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG 18:31:05 DEBUG "pod": "ds-cts-2" 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG "value": [ 18:31:05 DEBUG "0.005" 18:31:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:31:05 DEBUG }, 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:31:05 DEBUG 18:31:05 DEBUG ] 18:31:05 DEBUG }, 18:31:05 DEBUG "value": [ 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG 18:31:05 DEBUG } 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG { 18:31:05 DEBUG 18:31:05 DEBUG ] 18:31:05 INFO [http_cmd]: http status code OK 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "15790.18548361614" 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG [...] 18:31:05 DEBUG } 18:31:05 DEBUG --- status code --- 18:31:05 DEBUG "0" 18:31:05 DEBUG ] 18:31:05 DEBUG "data": { 18:31:05 DEBUG 18:31:05 DEBUG } 18:31:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG 18:31:05 DEBUG --- http response --- 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG "result": [ 18:31:05 DEBUG 18:31:05 DEBUG ] 18:31:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:31:05 DEBUG } 18:31:05 DEBUG { 18:31:05 DEBUG "pod": "ds-cts-2" 18:31:05 DEBUG } 18:31:05 DEBUG ----- output ----- 18:31:05 DEBUG } 18:31:05 DEBUG "metric": { 18:31:05 DEBUG }, 18:31:05 DEBUG } 18:31:05 DEBUG { 18:31:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:31:05 DEBUG "value": [ 18:31:05 DEBUG "status": "success", 18:31:05 DEBUG }, 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "data": { 18:31:05 DEBUG "value": [ 18:31:05 DEBUG "647.1387493285304" 18:31:05 DEBUG "resultType": "vector", 18:31:05 DEBUG 18:31:05 DEBUG ] 18:31:05 DEBUG "result": [ 18:31:05 DEBUG 18:31:05 DEBUG } 18:31:05 DEBUG { 18:31:05 DEBUG 18:31:05 DEBUG ] 18:31:05 DEBUG "metric": { 18:31:05 DEBUG [...] 18:31:05 DEBUG } 18:31:05 DEBUG "instance": "10.142.0.107:9100" 18:31:05 DEBUG 18:31:05 DEBUG } 18:31:05 DEBUG }, 18:31:05 DEBUG 18:31:05 DEBUG "value": [ 18:31:05 DEBUG 18:31:05 DEBUG 18:31:05 DEBUG "pod": "ds-cts-2" 18:31:05 DEBUG 18:31:05 DEBUG }, 18:31:05 DEBUG 18:31:05 DEBUG "value": [ 18:31:05 DEBUG [...] 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG 18:31:05 DEBUG "0" 18:31:05 DEBUG 18:31:05 DEBUG ] 18:31:05 DEBUG 18:31:05 DEBUG } 18:31:05 DEBUG "instance": "10.142.0.43:9100" 18:31:05 DEBUG ] 18:31:05 DEBUG }, 18:31:05 DEBUG } 18:31:05 DEBUG "value": [ 18:31:05 DEBUG } 18:31:05 DEBUG 1713288665, 18:31:05 DEBUG "0.00036666666666652263" 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG ] 18:31:05 DEBUG } 18:31:05 DEBUG } 18:31:05 INFO 18:31:05 INFO [loop_until]: kubectl --namespace=xlou top node 18:31:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:31:05 INFO [loop_until]: OK (rc = 0) 18:31:05 DEBUG --- stdout --- 18:31:05 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 5077Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 63m 0% 10815Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 51m 0% 5412Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 2660Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 63m 0% 6125Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 13007Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2981Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 59m 0% 4834Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 60m 0% 5450Mi 4% gke-xlou-cdm-ds-0f601751-j488 56m 0% 10348Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 81m 0% 3233Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 81m 0% 2983Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 58m 0% 7328Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 68m 0% 5401Mi 9% 18:31:05 DEBUG --- stderr --- 18:31:05 DEBUG 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "ds-idrepo-2" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "ds-idrepo-0" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288725" 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "ds-idrepo-2" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "ds-cts-2" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "instance": "10.142.0.107:9100" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "instance": "10.142.0.43:9100" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "9176.8" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "instance": "10.142.0.107:9100" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "instance": "10.142.0.43:9100" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "19831598080" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "ds-idrepo-2" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "ds-idrepo-0" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713288725" 18:32:05 INFO [loop_until]: kubectl --namespace=xlou top pods 18:32:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "1667575808" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "ds-idrepo-2" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "ds-idrepo-0" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 ERROR [http_cmd]: ERROR 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 ERROR --- status code --- 18:32:05 INFO [loop_until]: OK (rc = 0) 18:32:05 DEBUG --- status code --- 18:32:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- stdout --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 ERROR --- http response --- 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 6m 3126Mi am-555f4db595-fz8sn 7m 3182Mi am-555f4db595-kzxsp 7m 1121Mi ds-cts-0 6m 8290Mi ds-cts-1 5m 8305Mi ds-cts-2 4m 8446Mi ds-idrepo-0 8m 5029Mi ds-idrepo-1 8m 2779Mi ds-idrepo-2 8m 2858Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1042Mi idm-b85f9d978-rl76f 4m 1226Mi lodemon-69f8bd86c8-8fxnk 1m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 1497Mi 18:32:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:32:05 DEBUG --- stderr --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:32:05 DEBUG 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0.029413653580199418" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "2" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "instance": "10.142.0.107:9100" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "instance": "10.142.0.43:9100" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "16233.03333333333" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "ds-idrepo-2" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "ds-idrepo-0" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "ds-idrepo-2" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "ds-idrepo-0" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "ds-idrepo-0" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "4310.192628102072" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "instance": "10.142.0.107:9100" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "instance": "10.142.0.43:9100" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0.0009999999999999861" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "ds-idrepo-2" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "ds-idrepo-0" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288725" 18:32:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288725" 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "ds-idrepo-0" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0.007443015404522091" 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "data": { 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "result": [ 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG { 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:32:05 DEBUG }, 18:32:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 INFO [http_cmd]: http status code OK 18:32:05 DEBUG 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG --- status code --- 18:32:05 DEBUG 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:32:05 DEBUG 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:32:05 DEBUG 18:32:05 DEBUG --- http response --- 18:32:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:32:05 DEBUG }, 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:05 DEBUG "pod": "ds-idrepo-0" 18:32:05 DEBUG }, 18:32:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:32:05 DEBUG "value": [ 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG ----- output ----- 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG { 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG { 18:32:05 DEBUG "0" 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0.005" 18:32:05 DEBUG "status": "success", 18:32:05 DEBUG ] 18:32:05 DEBUG "data": { 18:32:05 DEBUG "2873.461752068048" 18:32:05 DEBUG ] 18:32:05 DEBUG "data": { 18:32:05 DEBUG } 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG "resultType": "vector", 18:32:05 DEBUG ] 18:32:05 DEBUG "result": [ 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG "result": [ 18:32:05 DEBUG } 18:32:05 DEBUG { 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG { 18:32:05 DEBUG } 18:32:05 DEBUG "metric": { 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 DEBUG "metric": { 18:32:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:32:05 DEBUG } 18:32:05 DEBUG "instance": "10.142.0.107:9100" 18:32:05 DEBUG }, 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG [...] 18:32:05 DEBUG [...] 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG 18:32:05 DEBUG "pod": "ds-idrepo-0" 18:32:05 DEBUG "instance": "10.142.0.43:9100" 18:32:05 DEBUG }, 18:32:05 DEBUG }, 18:32:05 DEBUG "value": [ 18:32:05 DEBUG "value": [ 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG 1713288725, 18:32:05 DEBUG "0" 18:32:05 DEBUG "0.0006000000000000227" 18:32:05 DEBUG ] 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 DEBUG ] 18:32:05 DEBUG ] 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 DEBUG } 18:32:05 INFO 18:32:05 INFO [loop_until]: kubectl --namespace=xlou top node 18:32:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:32:06 INFO [loop_until]: OK (rc = 0) 18:32:06 DEBUG --- stdout --- 18:32:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5081Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 61m 0% 10816Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 5400Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 2664Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 62m 0% 6133Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 148m 0% 13001Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2996Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 56m 0% 4901Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 61m 0% 5451Mi 4% gke-xlou-cdm-ds-0f601751-j488 57m 0% 10348Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 82m 0% 3230Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 85m 0% 2980Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 59m 0% 7329Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 71m 0% 5402Mi 9% 18:32:06 DEBUG --- stderr --- 18:32:06 DEBUG 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288785" 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "ds-idrepo-2" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "ds-cts-2" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "instance": "10.142.0.107:9100" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "instance": "10.142.0.43:9100" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "10669.066666666668" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "instance": "10.142.0.107:9100" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "instance": "10.142.0.43:9100" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "19832335360" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "ds-idrepo-2" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "ds-idrepo-0" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713288785" 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "ds-idrepo-2" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "ds-idrepo-0" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "1668169728" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "ds-idrepo-2" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "ds-idrepo-0" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713288785" 18:33:05 ERROR [http_cmd]: ERROR 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 ERROR --- status code --- 18:33:05 DEBUG --- status code --- 18:33:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 ERROR --- http response --- 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:33:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "ds-idrepo-2" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "ds-idrepo-0" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713288785" 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0.027522404689291468" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO 18:33:05 INFO [loop_until]: kubectl --namespace=xlou top pods 18:33:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "2" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:33:05 DEBUG }, 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 DEBUG "value": [ 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG "0" 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG ] 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG } 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:05 DEBUG ] 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG } 18:33:05 DEBUG { 18:33:05 DEBUG } 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "instance": "10.142.0.107:9100" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "instance": "10.142.0.43:9100" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "16107.499999999996" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "ds-idrepo-2" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "ds-idrepo-0" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [loop_until]: OK (rc = 0) 18:33:05 DEBUG --- stdout --- 18:33:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 17m 3233Mi am-555f4db595-fz8sn 13m 3194Mi am-555f4db595-kzxsp 4m 1120Mi ds-cts-0 5m 8290Mi ds-cts-1 4m 8305Mi ds-cts-2 5m 8446Mi ds-idrepo-0 39m 5029Mi ds-idrepo-1 8m 2779Mi ds-idrepo-2 8m 2859Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1042Mi idm-b85f9d978-rl76f 4m 1226Mi lodemon-69f8bd86c8-8fxnk 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 1497Mi 18:33:05 DEBUG --- stderr --- 18:33:05 DEBUG 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "ds-idrepo-2" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "ds-idrepo-0" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "ds-cts-1" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "instance": "10.142.0.107:9100" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "instance": "10.142.0.43:9100" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0.0003333333333333262" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "ds-cts-1" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713288785" 18:33:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288785" 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "ds-cts-1" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0.00002201039287725753" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG { 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG "data": { 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "pod": "ds-cts-1" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG [...] 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:33:05 DEBUG }, 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG "0" 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG --- http response --- 18:33:05 INFO [http_cmd]: http status code OK 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:33:05 DEBUG --- status code --- 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:33:05 DEBUG { 18:33:05 DEBUG --- http response --- 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "data": { 18:33:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG ----- output ----- 18:33:05 DEBUG "result": [ 18:33:05 DEBUG { 18:33:05 DEBUG { 18:33:05 DEBUG "status": "success", 18:33:05 DEBUG "metric": { 18:33:05 DEBUG "data": { 18:33:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:33:05 DEBUG "resultType": "vector", 18:33:05 DEBUG }, 18:33:05 DEBUG "result": [ 18:33:05 DEBUG "value": [ 18:33:05 DEBUG { 18:33:05 DEBUG 18:33:05 DEBUG "metric": { 18:33:05 DEBUG 18:33:05 DEBUG "instance": "10.142.0.107:9100" 18:33:05 DEBUG 18:33:05 DEBUG }, 18:33:05 DEBUG [...] 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG 18:33:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:33:05 DEBUG [...] 18:33:05 DEBUG }, 18:33:05 DEBUG 18:33:05 DEBUG "value": [ 18:33:05 DEBUG 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG 18:33:05 DEBUG "0.005" 18:33:05 DEBUG "instance": "10.142.0.43:9100" 18:33:05 DEBUG ] 18:33:05 DEBUG }, 18:33:05 DEBUG } 18:33:05 DEBUG "value": [ 18:33:05 DEBUG ] 18:33:05 DEBUG 1713288785, 18:33:05 DEBUG } 18:33:05 DEBUG "0.00043333333333303397" 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG ] 18:33:05 DEBUG } 18:33:05 DEBUG } 18:33:06 INFO 18:33:06 INFO [loop_until]: kubectl --namespace=xlou top node 18:33:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:33:06 INFO [loop_until]: OK (rc = 0) 18:33:06 DEBUG --- stdout --- 18:33:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5085Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 10818Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 5411Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 2654Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 61m 0% 6133Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 130m 0% 13004Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2995Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 66m 0% 4961Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 59m 0% 5454Mi 4% gke-xlou-cdm-ds-0f601751-j488 55m 0% 10347Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 78m 0% 3224Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 88m 0% 3000Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 102m 0% 7324Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 68m 0% 5402Mi 9% 18:33:06 DEBUG --- stderr --- 18:33:06 DEBUG 18:33:14 WARNING File /tmp/lodemon_data-20240416_182702/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [16/Apr/2024 18:33:14] "GET /monitoring/average?start_time=20240416_173007&stop_time=20240416_173113 HTTP/1.1" 200 - 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288845" 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "ds-idrepo-2" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "ds-cts-2" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "instance": "10.142.0.107:9100" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "instance": "10.142.0.43:9100" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "9253.533333333333" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "ds-idrepo-2" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "ds-idrepo-0" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "ds-idrepo-2" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "ds-idrepo-0" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288845" 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "instance": "10.142.0.107:9100" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "instance": "10.142.0.43:9100" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "19833246720" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "ds-idrepo-2" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "ds-idrepo-0" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713288845" 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713288845" 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "ds-idrepo-2" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "ds-idrepo-0" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG ] 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG } 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG } 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:05 ERROR [http_cmd]: ERROR 18:34:05 DEBUG ----- output ----- 18:34:05 ERROR --- status code --- 18:34:05 DEBUG { 18:34:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:34:05 DEBUG "status": "success", 18:34:05 ERROR --- http response --- 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "1668321280" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:34:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713288845" 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0.026547999204841313" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288845" 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "instance": "10.142.0.107:9100" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "instance": "10.142.0.43:9100" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "16281.333333333332" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "ds-idrepo-2" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "ds-idrepo-0" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 INFO [loop_until]: kubectl --namespace=xlou top pods 18:34:05 DEBUG --- status code --- 18:34:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "ds-idrepo-2" 18:34:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:34:05 DEBUG }, 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "ds-idrepo-0" 18:34:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:34:05 DEBUG }, 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0" 18:34:05 DEBUG "2" 18:34:05 DEBUG ] 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288845" 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "instance": "10.142.0.107:9100" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "instance": "10.142.0.43:9100" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288845" 18:34:05 INFO [loop_until]: OK (rc = 0) 18:34:05 DEBUG --- stdout --- 18:34:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 10m 3234Mi am-555f4db595-fz8sn 6m 3204Mi am-555f4db595-kzxsp 11m 1121Mi ds-cts-0 5m 8290Mi ds-cts-1 5m 8306Mi ds-cts-2 4m 8446Mi ds-idrepo-0 9m 5028Mi ds-idrepo-1 8m 2779Mi ds-idrepo-2 8m 2858Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1041Mi idm-b85f9d978-rl76f 4m 1226Mi lodemon-69f8bd86c8-8fxnk 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 1497Mi 18:34:05 DEBUG --- stderr --- 18:34:05 DEBUG 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "ds-cts-1" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0.003438010967939617" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "ds-cts-1" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "551.2229586515493" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "instance": "10.142.0.107:9100" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "instance": "10.142.0.43:9100" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0.00040000000000001514" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0.005" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "ds-cts-1" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288845" 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "0" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:05 INFO [http_cmd]: http status code OK 18:34:05 DEBUG --- status code --- 18:34:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:34:05 DEBUG --- http response --- 18:34:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:34:05 DEBUG ----- output ----- 18:34:05 DEBUG { 18:34:05 DEBUG "status": "success", 18:34:05 DEBUG "data": { 18:34:05 DEBUG "resultType": "vector", 18:34:05 DEBUG "result": [ 18:34:05 DEBUG { 18:34:05 DEBUG "metric": { 18:34:05 DEBUG "pod": "ds-cts-1" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG [...] 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG 18:34:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:34:05 DEBUG }, 18:34:05 DEBUG "value": [ 18:34:05 DEBUG 1713288845, 18:34:05 DEBUG "13091.545267974298" 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG ] 18:34:05 DEBUG } 18:34:05 DEBUG } 18:34:06 INFO 18:34:06 INFO [loop_until]: kubectl --namespace=xlou top node 18:34:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:34:06 INFO [loop_until]: OK (rc = 0) 18:34:06 DEBUG --- stdout --- 18:34:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5094Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 10818Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 52m 0% 5417Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 2655Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 63m 0% 6141Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 13014Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2992Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 60m 0% 4987Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 60m 0% 5448Mi 4% gke-xlou-cdm-ds-0f601751-j488 57m 0% 10340Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 77m 0% 3232Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 89m 0% 2999Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 59m 0% 7337Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 69m 0% 5401Mi 9% 18:34:06 DEBUG --- stderr --- 18:34:06 DEBUG 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "ds-idrepo-2" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "ds-cts-2" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288905" 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "instance": "10.142.0.107:9100" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "instance": "10.142.0.43:9100" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "8848.533333333333" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "ds-idrepo-2" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "ds-idrepo-0" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "ds-idrepo-2" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "ds-idrepo-0" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713288905" 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "ds-idrepo-2" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "ds-idrepo-0" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 ERROR [http_cmd]: ERROR 18:35:05 ERROR --- status code --- 18:35:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:35:05 ERROR --- http response --- 18:35:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:35:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "1665662976" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "instance": "10.142.0.107:9100" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "instance": "10.142.0.43:9100" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "19833940992" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "ds-idrepo-2" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "ds-idrepo-0" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0.024828138427129817" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "instance": "10.142.0.107:9100" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "instance": "10.142.0.43:9100" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "15973.399999999998" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "2" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288905" 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "ds-idrepo-2" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "ds-idrepo-0" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "instance": "10.142.0.107:9100" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "instance": "10.142.0.43:9100" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO 18:35:05 INFO [loop_until]: kubectl --namespace=xlou top pods 18:35:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0.007370712794512334" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "ds-idrepo-2" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "ds-idrepo-0" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "instance": "10.142.0.107:9100" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "instance": "10.142.0.43:9100" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0.00026666666666651885" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0.005" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [loop_until]: OK (rc = 0) 18:35:05 DEBUG --- stdout --- 18:35:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 7m 3240Mi am-555f4db595-fz8sn 4m 3213Mi am-555f4db595-kzxsp 7m 1121Mi ds-cts-0 5m 8290Mi ds-cts-1 5m 8305Mi ds-cts-2 4m 8446Mi ds-idrepo-0 8m 5029Mi ds-idrepo-1 8m 2779Mi ds-idrepo-2 8m 2858Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1041Mi idm-b85f9d978-rl76f 3m 1225Mi lodemon-69f8bd86c8-8fxnk 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 1497Mi 18:35:05 DEBUG --- stderr --- 18:35:05 DEBUG 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "542.9300460615701" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288905" 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "0" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:05 INFO [http_cmd]: http status code OK 18:35:05 DEBUG --- status code --- 18:35:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:35:05 DEBUG --- http response --- 18:35:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:35:05 DEBUG ----- output ----- 18:35:05 DEBUG { 18:35:05 DEBUG "status": "success", 18:35:05 DEBUG "data": { 18:35:05 DEBUG "resultType": "vector", 18:35:05 DEBUG "result": [ 18:35:05 DEBUG { 18:35:05 DEBUG "metric": { 18:35:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG [...] 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG 18:35:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:35:05 DEBUG }, 18:35:05 DEBUG "value": [ 18:35:05 DEBUG 1713288905, 18:35:05 DEBUG "30675.547602478713" 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG ] 18:35:05 DEBUG } 18:35:05 DEBUG } 18:35:06 INFO 18:35:06 INFO [loop_until]: kubectl --namespace=xlou top node 18:35:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:35:06 INFO [loop_until]: OK (rc = 0) 18:35:06 DEBUG --- stdout --- 18:35:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5088Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 10825Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 5431Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 2644Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 62m 0% 6150Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 12998Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2988Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 58m 0% 5004Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 59m 0% 5444Mi 4% gke-xlou-cdm-ds-0f601751-j488 55m 0% 10339Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 80m 0% 3231Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 88m 0% 2977Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 61m 0% 7342Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 68m 0% 5400Mi 9% 18:35:06 DEBUG --- stderr --- 18:35:06 DEBUG 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "pod": "ds-idrepo-2" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "pod": "ds-cts-2" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "0" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288965" 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "instance": "10.142.0.107:9100" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "instance": "10.142.0.43:9100" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "10596.466666666665" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "pod": "ds-idrepo-2" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "pod": "ds-idrepo-0" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "0" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "pod": "ds-idrepo-2" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "pod": "ds-idrepo-0" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "0" 18:36:05 DEBUG ] 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:05 DEBUG } 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG ] 18:36:05 DEBUG { 18:36:05 DEBUG } 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG } 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "0" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713288965" 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713288965" 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "pod": "ds-idrepo-2" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "pod": "ds-idrepo-0" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "0" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "1666375680" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 ERROR [http_cmd]: ERROR 18:36:05 ERROR --- status code --- 18:36:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:36:05 ERROR --- http response --- 18:36:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:36:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "pod": "ds-idrepo-2" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "pod": "ds-idrepo-0" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "0" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288965" 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG { 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "0" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "instance": "10.142.0.107:9100" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "instance": "10.142.0.43:9100" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "19834459136" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713288965" 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "0.032456877277704445" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "0" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288965" 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288965" 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "instance": "10.142.0.107:9100" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "instance": "10.142.0.43:9100" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "16164.266666666665" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "instance": "10.142.0.107:9100" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "instance": "10.142.0.43:9100" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "0.0006666666666667116" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "2" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO 18:36:05 INFO [loop_until]: kubectl --namespace=xlou top pods 18:36:05 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "pod": "ds-idrepo-2" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "pod": "ds-idrepo-0" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "0" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713288965" 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "data": { 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "result": [ 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG { 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:36:05 DEBUG "metric": { 18:36:05 DEBUG "pod": "ds-idrepo-2" 18:36:05 DEBUG }, 18:36:05 DEBUG "pod": "ds-cts-1" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG }, 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 18:36:05 DEBUG "value": [ 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG { 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "status": "success", 18:36:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:36:05 DEBUG 18:36:05 DEBUG "pod": "ds-idrepo-0" 18:36:05 DEBUG "data": { 18:36:05 DEBUG }, 18:36:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:36:05 DEBUG }, 18:36:05 DEBUG "resultType": "vector", 18:36:05 DEBUG "value": [ 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG "result": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG { 18:36:05 DEBUG "0.005" 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "0" 18:36:05 DEBUG "metric": { 18:36:05 DEBUG ] 18:36:05 DEBUG "0.00001834165220672694" 18:36:05 DEBUG ] 18:36:05 DEBUG "instance": "10.142.0.107:9100" 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG }, 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG "value": [ 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 DEBUG 18:36:05 DEBUG } 18:36:05 DEBUG 18:36:05 DEBUG [...] 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG 18:36:05 DEBUG "instance": "10.142.0.43:9100" 18:36:05 DEBUG }, 18:36:05 DEBUG "value": [ 18:36:05 DEBUG 1713288965, 18:36:05 DEBUG "0.0004666666666665264" 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG ] 18:36:05 DEBUG } 18:36:05 DEBUG } 18:36:05 INFO [loop_until]: OK (rc = 0) 18:36:05 DEBUG --- stdout --- 18:36:05 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 5m 3241Mi am-555f4db595-fz8sn 12m 3219Mi am-555f4db595-kzxsp 6m 1126Mi ds-cts-0 5m 8290Mi ds-cts-1 5m 8306Mi ds-cts-2 4m 8447Mi ds-idrepo-0 8m 5029Mi ds-idrepo-1 8m 2778Mi ds-idrepo-2 7m 2858Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1041Mi idm-b85f9d978-rl76f 4m 1225Mi lodemon-69f8bd86c8-8fxnk 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 1497Mi 18:36:05 DEBUG --- stderr --- 18:36:05 DEBUG 18:36:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288965" 18:36:05 INFO [http_cmd]: http status code OK 18:36:05 DEBUG --- status code --- 18:36:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:05 DEBUG --- http response --- 18:36:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:05 DEBUG ----- output ----- 18:36:05 DEBUG { 18:36:05 DEBUG "status": "success", 18:36:06 DEBUG "data": { 18:36:06 DEBUG "resultType": "vector", 18:36:06 DEBUG "result": [ 18:36:06 DEBUG { 18:36:06 DEBUG "metric": { 18:36:06 DEBUG "pod": "ds-cts-1" 18:36:06 DEBUG }, 18:36:06 DEBUG "value": [ 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG [...] 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:36:06 DEBUG }, 18:36:06 DEBUG "value": [ 18:36:06 DEBUG 1713288965, 18:36:06 DEBUG "0" 18:36:06 DEBUG ] 18:36:06 DEBUG } 18:36:06 DEBUG ] 18:36:06 DEBUG } 18:36:06 DEBUG } 18:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288966" 18:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288966" 18:36:06 INFO [http_cmd]: http status code OK 18:36:06 DEBUG --- status code --- 18:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:06 DEBUG --- http response --- 18:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:36:06 DEBUG ----- output ----- 18:36:06 DEBUG { 18:36:06 DEBUG "status": "success", 18:36:06 DEBUG "data": { 18:36:06 DEBUG "resultType": "vector", 18:36:06 DEBUG "result": [ 18:36:06 DEBUG { 18:36:06 DEBUG "metric": { 18:36:06 DEBUG "pod": "ds-cts-1" 18:36:06 DEBUG }, 18:36:06 DEBUG "value": [ 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG [...] 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:36:06 DEBUG }, 18:36:06 DEBUG "value": [ 18:36:06 DEBUG 1713288966, 18:36:06 DEBUG "3266.363002640513" 18:36:06 DEBUG ] 18:36:06 DEBUG } 18:36:06 DEBUG ] 18:36:06 DEBUG } 18:36:06 DEBUG } 18:36:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713288966" 18:36:06 INFO [http_cmd]: http status code OK 18:36:06 DEBUG --- status code --- 18:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:06 DEBUG --- http response --- 18:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:36:06 DEBUG ----- output ----- 18:36:06 DEBUG { 18:36:06 DEBUG "status": "success", 18:36:06 DEBUG "data": { 18:36:06 DEBUG "resultType": "vector", 18:36:06 DEBUG "result": [ 18:36:06 DEBUG { 18:36:06 DEBUG "metric": { 18:36:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:36:06 DEBUG }, 18:36:06 DEBUG "value": [ 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG [...] 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:36:06 DEBUG }, 18:36:06 DEBUG "value": [ 18:36:06 DEBUG 1713288966, 18:36:06 DEBUG "0" 18:36:06 DEBUG ] 18:36:06 DEBUG } 18:36:06 DEBUG ] 18:36:06 DEBUG } 18:36:06 DEBUG } 18:36:06 INFO [http_cmd]: http status code OK 18:36:06 DEBUG --- status code --- 18:36:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:36:06 DEBUG --- http response --- 18:36:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:36:06 DEBUG ----- output ----- 18:36:06 DEBUG { 18:36:06 DEBUG "status": "success", 18:36:06 DEBUG "data": { 18:36:06 DEBUG "resultType": "vector", 18:36:06 DEBUG "result": [ 18:36:06 DEBUG { 18:36:06 DEBUG "metric": { 18:36:06 DEBUG "pod": "ds-cts-1" 18:36:06 DEBUG }, 18:36:06 DEBUG "value": [ 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG [...] 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG 18:36:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:36:06 DEBUG }, 18:36:06 DEBUG "value": [ 18:36:06 DEBUG 1713288966, 18:36:06 DEBUG "0" 18:36:06 DEBUG ] 18:36:06 DEBUG } 18:36:06 DEBUG ] 18:36:06 DEBUG } 18:36:06 DEBUG } 18:36:06 INFO 18:36:06 INFO [loop_until]: kubectl --namespace=xlou top node 18:36:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:36:06 INFO [loop_until]: OK (rc = 0) 18:36:06 DEBUG --- stdout --- 18:36:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5082Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 10836Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 5434Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 2633Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 63m 0% 6153Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 147m 0% 12997Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2983Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 56m 0% 5007Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 60m 0% 5453Mi 4% gke-xlou-cdm-ds-0f601751-j488 56m 0% 10347Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 79m 0% 3231Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 84m 0% 2995Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 62m 0% 7334Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 69m 0% 5403Mi 9% 18:36:06 DEBUG --- stderr --- 18:36:06 DEBUG 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "pod": "ds-idrepo-2" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "pod": "ds-cts-2" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "0" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "pod": "ds-idrepo-2" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "pod": "ds-idrepo-0" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "0" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "pod": "ds-idrepo-2" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "pod": "ds-idrepo-0" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "0" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289025" 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "instance": "10.142.0.107:9100" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "instance": "10.142.0.43:9100" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "9312.666666666666" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "0" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "pod": "ds-idrepo-2" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "pod": "ds-idrepo-0" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "0" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713289025" 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713289025" 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "0" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "1666813952" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 ERROR [http_cmd]: ERROR 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 ERROR --- status code --- 18:37:05 DEBUG --- status code --- 18:37:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 ERROR --- http response --- 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:37:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "pod": "ds-idrepo-2" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "pod": "ds-idrepo-0" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "0" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289025" 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "instance": "10.142.0.107:9100" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "instance": "10.142.0.43:9100" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "19835087872" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713289025" 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "0.032240051523585556" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289025" 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "0" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "instance": "10.142.0.107:9100" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "instance": "10.142.0.43:9100" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "16216.699999999997" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289025" 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "2" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "instance": "10.142.0.107:9100" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "instance": "10.142.0.43:9100" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "0" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "pod": "ds-idrepo-2" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "pod": "ds-idrepo-0" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "0" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:05 INFO [http_cmd]: http status code OK 18:37:05 DEBUG --- status code --- 18:37:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:05 DEBUG --- http response --- 18:37:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:37:05 DEBUG ----- output ----- 18:37:05 DEBUG { 18:37:05 DEBUG "status": "success", 18:37:05 DEBUG "data": { 18:37:05 DEBUG "resultType": "vector", 18:37:05 DEBUG "result": [ 18:37:05 DEBUG { 18:37:05 DEBUG "metric": { 18:37:05 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG [...] 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG 18:37:05 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:37:05 DEBUG }, 18:37:05 DEBUG "value": [ 18:37:05 DEBUG 1713289025, 18:37:05 DEBUG "0.0033458829330107074" 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG ] 18:37:05 DEBUG } 18:37:05 DEBUG } 18:37:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713289025" 18:37:06 INFO [http_cmd]: http status code OK 18:37:06 DEBUG --- status code --- 18:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:06 DEBUG --- http response --- 18:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:06 DEBUG ----- output ----- 18:37:06 DEBUG { 18:37:06 DEBUG "status": "success", 18:37:06 DEBUG "data": { 18:37:06 DEBUG "resultType": "vector", 18:37:06 DEBUG "result": [ 18:37:06 DEBUG { 18:37:06 DEBUG "metric": { 18:37:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG [...] 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 1713289025, 18:37:06 DEBUG "0.005" 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG } 18:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289026" 18:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289026" 18:37:06 INFO [http_cmd]: http status code OK 18:37:06 INFO [http_cmd]: http status code OK 18:37:06 DEBUG --- status code --- 18:37:06 DEBUG --- status code --- 18:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:06 DEBUG --- http response --- 18:37:06 DEBUG --- http response --- 18:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:06 DEBUG ----- output ----- 18:37:06 DEBUG { 18:37:06 DEBUG "status": "success", 18:37:06 DEBUG "data": { 18:37:06 DEBUG "resultType": "vector", 18:37:06 DEBUG "result": [ 18:37:06 DEBUG { 18:37:06 DEBUG "metric": { 18:37:06 DEBUG "pod": "ds-idrepo-2" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG [...] 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG "pod": "ds-idrepo-0" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 1713289026, 18:37:06 DEBUG "0" 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG } 18:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:37:06 DEBUG ----- output ----- 18:37:06 DEBUG { 18:37:06 DEBUG "status": "success", 18:37:06 DEBUG "data": { 18:37:06 DEBUG "resultType": "vector", 18:37:06 DEBUG "result": [ 18:37:06 DEBUG { 18:37:06 DEBUG "metric": { 18:37:06 DEBUG "instance": "10.142.0.107:9100" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG [...] 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG "instance": "10.142.0.43:9100" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 1713289026, 18:37:06 DEBUG "0.00033333333333350387" 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG } 18:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289026" 18:37:06 INFO [http_cmd]: http status code OK 18:37:06 DEBUG --- status code --- 18:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:06 DEBUG --- http response --- 18:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 18:37:06 DEBUG ----- output ----- 18:37:06 DEBUG { 18:37:06 DEBUG "status": "success", 18:37:06 DEBUG "data": { 18:37:06 DEBUG "resultType": "vector", 18:37:06 DEBUG "result": [ 18:37:06 DEBUG { 18:37:06 DEBUG "metric": { 18:37:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG [...] 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 1713289026, 18:37:06 DEBUG "4616.1865182785095" 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG } 18:37:06 INFO 18:37:06 INFO [loop_until]: kubectl --namespace=xlou top pods 18:37:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289026" 18:37:06 INFO [http_cmd]: http status code OK 18:37:06 DEBUG --- status code --- 18:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:06 DEBUG --- http response --- 18:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:37:06 DEBUG ----- output ----- 18:37:06 DEBUG { 18:37:06 DEBUG "status": "success", 18:37:06 DEBUG "data": { 18:37:06 DEBUG "resultType": "vector", 18:37:06 DEBUG "result": [ 18:37:06 DEBUG { 18:37:06 DEBUG "metric": { 18:37:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG [...] 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 1713289026, 18:37:06 DEBUG "0" 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG } 18:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289026" 18:37:06 INFO [http_cmd]: http status code OK 18:37:06 DEBUG --- status code --- 18:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:06 DEBUG --- http response --- 18:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 18:37:06 DEBUG ----- output ----- 18:37:06 DEBUG { 18:37:06 DEBUG "status": "success", 18:37:06 DEBUG "data": { 18:37:06 DEBUG "resultType": "vector", 18:37:06 DEBUG "result": [ 18:37:06 DEBUG { 18:37:06 DEBUG "metric": { 18:37:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG [...] 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 1713289026, 18:37:06 DEBUG "577.0233147848137" 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG } 18:37:06 INFO [loop_until]: OK (rc = 0) 18:37:06 DEBUG --- stdout --- 18:37:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 5m 3244Mi am-555f4db595-fz8sn 4m 3219Mi am-555f4db595-kzxsp 6m 1125Mi ds-cts-0 5m 8290Mi ds-cts-1 5m 8306Mi ds-cts-2 5m 8446Mi ds-idrepo-0 8m 5029Mi ds-idrepo-1 8m 2779Mi ds-idrepo-2 8m 2858Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1042Mi idm-b85f9d978-rl76f 3m 1225Mi lodemon-69f8bd86c8-8fxnk 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 5Mi overseer-0-5584f6c4db-bcn5q 1m 1497Mi 18:37:06 DEBUG --- stderr --- 18:37:06 DEBUG 18:37:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289026" 18:37:06 INFO [http_cmd]: http status code OK 18:37:06 DEBUG --- status code --- 18:37:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:37:06 DEBUG --- http response --- 18:37:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 18:37:06 DEBUG ----- output ----- 18:37:06 DEBUG { 18:37:06 DEBUG "status": "success", 18:37:06 DEBUG "data": { 18:37:06 DEBUG "resultType": "vector", 18:37:06 DEBUG "result": [ 18:37:06 DEBUG { 18:37:06 DEBUG "metric": { 18:37:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG [...] 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG 18:37:06 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:37:06 DEBUG }, 18:37:06 DEBUG "value": [ 18:37:06 DEBUG 1713289026, 18:37:06 DEBUG "0" 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG ] 18:37:06 DEBUG } 18:37:06 DEBUG } 18:37:06 INFO 18:37:06 INFO [loop_until]: kubectl --namespace=xlou top node 18:37:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:37:06 INFO [loop_until]: OK (rc = 0) 18:37:06 DEBUG --- stdout --- 18:37:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5087Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 66m 0% 10829Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 5408Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 2646Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 61m 0% 6148Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 147m 0% 13012Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2975Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 55m 0% 5003Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 60m 0% 5447Mi 4% gke-xlou-cdm-ds-0f601751-j488 56m 0% 10339Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 78m 0% 3227Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 83m 0% 2979Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 60m 0% 7329Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 69m 0% 5407Mi 9% 18:37:06 DEBUG --- stderr --- 18:37:06 DEBUG 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289085" 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289085" 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713289085" 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "pod": "ds-idrepo-2" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "pod": "ds-idrepo-0" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "0" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "pod": "ds-idrepo-2" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "pod": "ds-cts-2" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "0" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289085" 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289085" 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "pod": "ds-idrepo-2" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "pod": "ds-idrepo-0" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "0" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289085" 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713289085" 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "instance": "10.142.0.107:9100" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "instance": "10.142.0.43:9100" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "10559.133333333333" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "1669177344" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289085" 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289085" 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "0" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "pod": "ds-idrepo-2" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "pod": "ds-idrepo-0" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "0" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289085" 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713289085" 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289085" 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "0" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 ERROR [http_cmd]: ERROR 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 ERROR --- status code --- 18:38:05 DEBUG --- status code --- 18:38:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 ERROR --- http response --- 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:38:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "pod": "ds-idrepo-2" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "pod": "ds-idrepo-0" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "0" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289085" 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "instance": "10.142.0.107:9100" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "instance": "10.142.0.43:9100" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "19835675648" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713289085" 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289085" 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "0.02863380594627417" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "instance": "10.142.0.107:9100" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "instance": "10.142.0.43:9100" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "16118.399999999996" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289085" 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289085" 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289085" 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "0" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "pod": "ds-idrepo-2" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "pod": "ds-idrepo-0" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "0" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: http status code OK 18:38:05 DEBUG --- status code --- 18:38:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:05 DEBUG --- http response --- 18:38:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:05 DEBUG ----- output ----- 18:38:05 DEBUG { 18:38:05 DEBUG "status": "success", 18:38:05 DEBUG "data": { 18:38:05 DEBUG "resultType": "vector", 18:38:05 DEBUG "result": [ 18:38:05 DEBUG { 18:38:05 DEBUG "metric": { 18:38:05 DEBUG "instance": "10.142.0.107:9100" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG [...] 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG 18:38:05 DEBUG "instance": "10.142.0.43:9100" 18:38:05 DEBUG }, 18:38:05 DEBUG "value": [ 18:38:05 DEBUG 1713289085, 18:38:05 DEBUG "0" 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG ] 18:38:05 DEBUG } 18:38:05 DEBUG } 18:38:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289085" 18:38:06 INFO [http_cmd]: http status code OK 18:38:06 DEBUG --- status code --- 18:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:06 DEBUG --- http response --- 18:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:38:06 DEBUG ----- output ----- 18:38:06 DEBUG { 18:38:06 DEBUG "status": "success", 18:38:06 DEBUG "data": { 18:38:06 DEBUG "resultType": "vector", 18:38:06 DEBUG "result": [ 18:38:06 DEBUG { 18:38:06 DEBUG "metric": { 18:38:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG [...] 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG "pod": "ds-cts-2" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 1713289085, 18:38:06 DEBUG "0" 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG } 18:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289086" 18:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289086" 18:38:06 INFO [http_cmd]: http status code OK 18:38:06 DEBUG --- status code --- 18:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:06 DEBUG --- http response --- 18:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:06 DEBUG ----- output ----- 18:38:06 DEBUG { 18:38:06 DEBUG "status": "success", 18:38:06 DEBUG "data": { 18:38:06 DEBUG "resultType": "vector", 18:38:06 DEBUG "result": [ 18:38:06 DEBUG { 18:38:06 DEBUG "metric": { 18:38:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG [...] 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 1713289086, 18:38:06 DEBUG "2" 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG } 18:38:06 INFO [http_cmd]: http status code OK 18:38:06 DEBUG --- status code --- 18:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:06 DEBUG --- http response --- 18:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:06 DEBUG ----- output ----- 18:38:06 DEBUG { 18:38:06 DEBUG "status": "success", 18:38:06 DEBUG "data": { 18:38:06 DEBUG "resultType": "vector", 18:38:06 DEBUG "result": [ 18:38:06 DEBUG { 18:38:06 DEBUG "metric": { 18:38:06 DEBUG "pod": "ds-idrepo-2" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG [...] 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG "pod": "ds-idrepo-0" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 1713289086, 18:38:06 DEBUG "0" 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG } 18:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289086" 18:38:06 INFO [http_cmd]: http status code OK 18:38:06 DEBUG --- status code --- 18:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:06 DEBUG --- http response --- 18:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:38:06 DEBUG ----- output ----- 18:38:06 DEBUG { 18:38:06 DEBUG "status": "success", 18:38:06 DEBUG "data": { 18:38:06 DEBUG "resultType": "vector", 18:38:06 DEBUG "result": [ 18:38:06 DEBUG { 18:38:06 DEBUG "metric": { 18:38:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG [...] 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG "pod": "ds-cts-2" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 1713289086, 18:38:06 DEBUG "0" 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG } 18:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289086" 18:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713289086" 18:38:06 INFO [http_cmd]: http status code OK 18:38:06 DEBUG --- status code --- 18:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:06 DEBUG --- http response --- 18:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:06 DEBUG ----- output ----- 18:38:06 DEBUG { 18:38:06 DEBUG "status": "success", 18:38:06 DEBUG "data": { 18:38:06 DEBUG "resultType": "vector", 18:38:06 DEBUG "result": [ 18:38:06 DEBUG { 18:38:06 DEBUG "metric": { 18:38:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG [...] 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 1713289086, 18:38:06 DEBUG "0" 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG } 18:38:06 INFO [http_cmd]: http status code OK 18:38:06 DEBUG --- status code --- 18:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:06 DEBUG --- http response --- 18:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:38:06 DEBUG ----- output ----- 18:38:06 DEBUG { 18:38:06 DEBUG "status": "success", 18:38:06 DEBUG "data": { 18:38:06 DEBUG "resultType": "vector", 18:38:06 DEBUG "result": [ 18:38:06 DEBUG { 18:38:06 DEBUG "metric": { 18:38:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG [...] 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 1713289086, 18:38:06 DEBUG "0.005" 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG } 18:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289086" 18:38:06 INFO [http_cmd]: http status code OK 18:38:06 DEBUG --- status code --- 18:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:06 DEBUG --- http response --- 18:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:38:06 DEBUG ----- output ----- 18:38:06 DEBUG { 18:38:06 DEBUG "status": "success", 18:38:06 DEBUG "data": { 18:38:06 DEBUG "resultType": "vector", 18:38:06 DEBUG "result": [ 18:38:06 DEBUG { 18:38:06 DEBUG "metric": { 18:38:06 DEBUG "instance": "10.142.0.107:9100" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG [...] 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG "instance": "10.142.0.43:9100" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 1713289086, 18:38:06 DEBUG "0.00036666666666652263" 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG } 18:38:06 INFO 18:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289086" 18:38:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289086" 18:38:06 INFO [loop_until]: kubectl --namespace=xlou top pods 18:38:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:38:06 INFO [http_cmd]: http status code OK 18:38:06 DEBUG --- status code --- 18:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:06 DEBUG --- http response --- 18:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 18:38:06 DEBUG ----- output ----- 18:38:06 DEBUG { 18:38:06 DEBUG "status": "success", 18:38:06 DEBUG "data": { 18:38:06 DEBUG "resultType": "vector", 18:38:06 DEBUG "result": [ 18:38:06 DEBUG { 18:38:06 DEBUG "metric": { 18:38:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG [...] 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 1713289086, 18:38:06 DEBUG "0" 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG } 18:38:06 INFO [http_cmd]: http status code OK 18:38:06 DEBUG --- status code --- 18:38:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:38:06 DEBUG --- http response --- 18:38:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 18:38:06 DEBUG ----- output ----- 18:38:06 DEBUG { 18:38:06 DEBUG "status": "success", 18:38:06 DEBUG "data": { 18:38:06 DEBUG "resultType": "vector", 18:38:06 DEBUG "result": [ 18:38:06 DEBUG { 18:38:06 DEBUG "metric": { 18:38:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG [...] 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG 18:38:06 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 18:38:06 DEBUG }, 18:38:06 DEBUG "value": [ 18:38:06 DEBUG 1713289086, 18:38:06 DEBUG "0" 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG ] 18:38:06 DEBUG } 18:38:06 DEBUG } 18:38:06 INFO [loop_until]: OK (rc = 0) 18:38:06 DEBUG --- stdout --- 18:38:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 11m 3247Mi am-555f4db595-fz8sn 11m 3221Mi am-555f4db595-kzxsp 5m 1120Mi ds-cts-0 5m 8289Mi ds-cts-1 4m 8306Mi ds-cts-2 4m 8446Mi ds-idrepo-0 8m 5029Mi ds-idrepo-1 8m 2779Mi ds-idrepo-2 7m 2857Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1041Mi idm-b85f9d978-rl76f 3m 1225Mi lodemon-69f8bd86c8-8fxnk 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 1497Mi 18:38:06 DEBUG --- stderr --- 18:38:06 DEBUG 18:38:06 INFO 18:38:06 INFO [loop_until]: kubectl --namespace=xlou top node 18:38:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:38:06 INFO [loop_until]: OK (rc = 0) 18:38:06 DEBUG --- stdout --- 18:38:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5081Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 68m 0% 10821Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 59m 0% 5423Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 2647Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 59m 0% 6145Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 129m 0% 13019Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2987Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 62m 0% 5005Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5452Mi 4% gke-xlou-cdm-ds-0f601751-j488 56m 0% 10347Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 81m 0% 3224Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 78m 0% 2976Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 62m 0% 7320Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 70m 0% 5399Mi 9% 18:38:06 DEBUG --- stderr --- 18:38:06 DEBUG 18:38:20 WARNING File /tmp/lodemon_data-20240416_182702/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [16/Apr/2024 18:38:20] "GET /monitoring/average?start_time=20240416_173514&stop_time=20240416_173620 HTTP/1.1" 200 - 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713289145" 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289145" 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289145" 18:39:05 INFO [http_cmd]: http status code OK 18:39:05 INFO [http_cmd]: http status code OK 18:39:05 DEBUG --- status code --- 18:39:05 DEBUG --- status code --- 18:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:05 DEBUG --- http response --- 18:39:05 DEBUG --- http response --- 18:39:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:39:05 DEBUG ----- output ----- 18:39:05 DEBUG { 18:39:05 DEBUG "status": "success", 18:39:05 DEBUG "data": { 18:39:05 DEBUG "resultType": "vector", 18:39:05 DEBUG "result": [ 18:39:05 DEBUG { 18:39:05 DEBUG "metric": { 18:39:05 DEBUG "pod": "ds-idrepo-2" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG [...] 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG "pod": "ds-cts-2" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 1713289145, 18:39:05 DEBUG "0" 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG } 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289145" 18:39:05 INFO [http_cmd]: http status code OK 18:39:05 DEBUG --- status code --- 18:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:05 DEBUG --- http response --- 18:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:39:05 DEBUG ----- output ----- 18:39:05 DEBUG { 18:39:05 DEBUG "status": "success", 18:39:05 DEBUG "data": { 18:39:05 DEBUG "resultType": "vector", 18:39:05 DEBUG "result": [ 18:39:05 DEBUG { 18:39:05 DEBUG "metric": { 18:39:05 DEBUG "pod": "ds-idrepo-2" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG [...] 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG "pod": "ds-idrepo-0" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 1713289145, 18:39:05 DEBUG "0" 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG } 18:39:05 INFO [http_cmd]: http status code OK 18:39:05 DEBUG --- status code --- 18:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:05 DEBUG --- http response --- 18:39:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289145" 18:39:05 INFO [http_cmd]: http status code OK 18:39:05 DEBUG --- status code --- 18:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:05 DEBUG --- http response --- 18:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:39:05 DEBUG ----- output ----- 18:39:05 DEBUG { 18:39:05 DEBUG "status": "success", 18:39:05 DEBUG "data": { 18:39:05 DEBUG "resultType": "vector", 18:39:05 DEBUG "result": [ 18:39:05 DEBUG { 18:39:05 DEBUG "metric": { 18:39:05 DEBUG "pod": "ds-idrepo-2" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG [...] 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG "pod": "ds-idrepo-0" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 1713289145, 18:39:05 DEBUG "0" 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG } 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713289145" 18:39:05 INFO [http_cmd]: http status code OK 18:39:05 DEBUG --- status code --- 18:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:05 DEBUG --- http response --- 18:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:05 DEBUG ----- output ----- 18:39:05 DEBUG { 18:39:05 DEBUG "status": "success", 18:39:05 DEBUG "data": { 18:39:05 DEBUG "resultType": "vector", 18:39:05 DEBUG "result": [ 18:39:05 DEBUG { 18:39:05 DEBUG "metric": { 18:39:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG [...] 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 1713289145, 18:39:05 DEBUG "1666732032" 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG } 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289145" 18:39:05 INFO [http_cmd]: http status code OK 18:39:05 DEBUG --- status code --- 18:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:05 DEBUG --- http response --- 18:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:05 DEBUG ----- output ----- 18:39:05 DEBUG { 18:39:05 DEBUG "status": "success", 18:39:05 DEBUG "data": { 18:39:05 DEBUG "resultType": "vector", 18:39:05 DEBUG "result": [ 18:39:05 DEBUG { 18:39:05 DEBUG "metric": { 18:39:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG [...] 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 1713289145, 18:39:05 DEBUG "0" 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG } 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289145" 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713289145" 18:39:05 INFO [http_cmd]: http status code OK 18:39:05 DEBUG --- status code --- 18:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:05 DEBUG --- http response --- 18:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:05 DEBUG ----- output ----- 18:39:05 DEBUG { 18:39:05 DEBUG "status": "success", 18:39:05 DEBUG "data": { 18:39:05 DEBUG "resultType": "vector", 18:39:05 DEBUG "result": [ 18:39:05 DEBUG { 18:39:05 DEBUG "metric": { 18:39:05 DEBUG "instance": "10.142.0.107:9100" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG [...] 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG "instance": "10.142.0.43:9100" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 1713289145, 18:39:05 DEBUG "8927.566666666668" 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG } 18:39:05 ERROR [http_cmd]: ERROR 18:39:05 ERROR --- status code --- 18:39:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:39:05 ERROR --- http response --- 18:39:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:39:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289145" 18:39:05 INFO [http_cmd]: http status code OK 18:39:05 DEBUG --- status code --- 18:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:05 DEBUG --- http response --- 18:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:05 DEBUG ----- output ----- 18:39:05 DEBUG { 18:39:05 DEBUG "status": "success", 18:39:05 DEBUG "data": { 18:39:05 DEBUG "resultType": "vector", 18:39:05 DEBUG "result": [ 18:39:05 DEBUG { 18:39:05 DEBUG "metric": { 18:39:05 DEBUG "pod": "ds-idrepo-2" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG [...] 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG "pod": "ds-idrepo-0" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 1713289145, 18:39:05 DEBUG "0" 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG } 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289145" 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289145" 18:39:05 INFO [http_cmd]: http status code OK 18:39:05 DEBUG --- status code --- 18:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:05 DEBUG --- http response --- 18:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:05 DEBUG ----- output ----- 18:39:05 DEBUG { 18:39:05 DEBUG "status": "success", 18:39:05 DEBUG "data": { 18:39:05 DEBUG "resultType": "vector", 18:39:05 DEBUG "result": [ 18:39:05 DEBUG { 18:39:05 DEBUG "metric": { 18:39:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG [...] 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 1713289145, 18:39:05 DEBUG "0" 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG } 18:39:05 INFO [http_cmd]: http status code OK 18:39:05 DEBUG --- status code --- 18:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:05 DEBUG --- http response --- 18:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:39:05 DEBUG ----- output ----- 18:39:05 DEBUG { 18:39:05 DEBUG "status": "success", 18:39:05 DEBUG "data": { 18:39:05 DEBUG "resultType": "vector", 18:39:05 DEBUG "result": [ 18:39:05 DEBUG { 18:39:05 DEBUG "metric": { 18:39:05 DEBUG "pod": "ds-idrepo-2" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG [...] 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG "pod": "ds-idrepo-0" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 1713289145, 18:39:05 DEBUG "0" 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG } 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289145" 18:39:05 INFO [http_cmd]: http status code OK 18:39:05 DEBUG --- status code --- 18:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:05 DEBUG --- http response --- 18:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:05 DEBUG ----- output ----- 18:39:05 DEBUG { 18:39:05 DEBUG "status": "success", 18:39:05 DEBUG "data": { 18:39:05 DEBUG "resultType": "vector", 18:39:05 DEBUG "result": [ 18:39:05 DEBUG { 18:39:05 DEBUG "metric": { 18:39:05 DEBUG "instance": "10.142.0.107:9100" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG [...] 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG "instance": "10.142.0.43:9100" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 1713289145, 18:39:05 DEBUG "15845.233333333332" 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG } 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289145" 18:39:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289145" 18:39:05 INFO [http_cmd]: http status code OK 18:39:05 DEBUG --- status code --- 18:39:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:05 DEBUG --- http response --- 18:39:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:05 DEBUG ----- output ----- 18:39:05 DEBUG { 18:39:05 DEBUG "status": "success", 18:39:05 DEBUG "data": { 18:39:05 DEBUG "resultType": "vector", 18:39:05 DEBUG "result": [ 18:39:05 DEBUG { 18:39:05 DEBUG "metric": { 18:39:05 DEBUG "instance": "10.142.0.107:9100" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG [...] 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG 18:39:05 DEBUG "instance": "10.142.0.43:9100" 18:39:05 DEBUG }, 18:39:05 DEBUG "value": [ 18:39:05 DEBUG 1713289145, 18:39:05 DEBUG "19836353536" 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG ] 18:39:05 DEBUG } 18:39:05 DEBUG } 18:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713289146" 18:39:06 INFO [http_cmd]: http status code OK 18:39:06 DEBUG --- status code --- 18:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:06 DEBUG --- http response --- 18:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:06 DEBUG ----- output ----- 18:39:06 DEBUG { 18:39:06 DEBUG "status": "success", 18:39:06 DEBUG "data": { 18:39:06 DEBUG "resultType": "vector", 18:39:06 DEBUG "result": [ 18:39:06 DEBUG { 18:39:06 DEBUG "metric": { 18:39:06 DEBUG "instance": "10.142.0.107:9100" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG [...] 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG "instance": "10.142.0.43:9100" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 1713289145, 18:39:06 DEBUG "0.000333333333333341" 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG } 18:39:06 INFO [http_cmd]: http status code OK 18:39:06 DEBUG --- status code --- 18:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:06 DEBUG --- http response --- 18:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:06 DEBUG ----- output ----- 18:39:06 DEBUG { 18:39:06 DEBUG "status": "success", 18:39:06 DEBUG "data": { 18:39:06 DEBUG "resultType": "vector", 18:39:06 DEBUG "result": [ 18:39:06 DEBUG { 18:39:06 DEBUG "metric": { 18:39:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG [...] 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 1713289146, 18:39:06 DEBUG "0.027646051566981026" 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG } 18:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289146" 18:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289146" 18:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289146" 18:39:06 INFO [http_cmd]: http status code OK 18:39:06 DEBUG --- status code --- 18:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:06 DEBUG --- http response --- 18:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:06 DEBUG ----- output ----- 18:39:06 DEBUG { 18:39:06 DEBUG "status": "success", 18:39:06 DEBUG "data": { 18:39:06 DEBUG "resultType": "vector", 18:39:06 DEBUG "result": [ 18:39:06 DEBUG { 18:39:06 DEBUG "metric": { 18:39:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG [...] 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 1713289146, 18:39:06 DEBUG "0" 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG } 18:39:06 INFO [http_cmd]: http status code OK 18:39:06 DEBUG --- status code --- 18:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:06 DEBUG --- http response --- 18:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:06 DEBUG ----- output ----- 18:39:06 DEBUG { 18:39:06 DEBUG "status": "success", 18:39:06 DEBUG "data": { 18:39:06 DEBUG "resultType": "vector", 18:39:06 DEBUG "result": [ 18:39:06 DEBUG { 18:39:06 DEBUG "metric": { 18:39:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG [...] 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 1713289146, 18:39:06 DEBUG "2" 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG } 18:39:06 INFO [http_cmd]: http status code OK 18:39:06 DEBUG --- status code --- 18:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:06 DEBUG --- http response --- 18:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:06 DEBUG ----- output ----- 18:39:06 DEBUG { 18:39:06 DEBUG "status": "success", 18:39:06 DEBUG "data": { 18:39:06 DEBUG "resultType": "vector", 18:39:06 DEBUG "result": [ 18:39:06 DEBUG { 18:39:06 DEBUG "metric": { 18:39:06 DEBUG "pod": "ds-idrepo-2" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG [...] 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG "pod": "ds-idrepo-0" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 1713289146, 18:39:06 DEBUG "0" 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG } 18:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289146" 18:39:06 INFO [http_cmd]: http status code OK 18:39:06 DEBUG --- status code --- 18:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:06 DEBUG --- http response --- 18:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:06 DEBUG ----- output ----- 18:39:06 DEBUG { 18:39:06 DEBUG "status": "success", 18:39:06 DEBUG "data": { 18:39:06 DEBUG "resultType": "vector", 18:39:06 DEBUG "result": [ 18:39:06 DEBUG { 18:39:06 DEBUG "metric": { 18:39:06 DEBUG "pod": "ds-cts-1" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG [...] 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 1713289146, 18:39:06 DEBUG "0.00002225322399537012" 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG } 18:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289146" 18:39:06 INFO [http_cmd]: http status code OK 18:39:06 DEBUG --- status code --- 18:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:06 DEBUG --- http response --- 18:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:06 DEBUG ----- output ----- 18:39:06 DEBUG { 18:39:06 DEBUG "status": "success", 18:39:06 DEBUG "data": { 18:39:06 DEBUG "resultType": "vector", 18:39:06 DEBUG "result": [ 18:39:06 DEBUG { 18:39:06 DEBUG "metric": { 18:39:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG [...] 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 1713289146, 18:39:06 DEBUG "0" 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG } 18:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289146" 18:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289146" 18:39:06 INFO [http_cmd]: http status code OK 18:39:06 DEBUG --- status code --- 18:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:06 DEBUG --- http response --- 18:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:06 DEBUG ----- output ----- 18:39:06 DEBUG { 18:39:06 DEBUG "status": "success", 18:39:06 DEBUG "data": { 18:39:06 DEBUG "resultType": "vector", 18:39:06 DEBUG "result": [ 18:39:06 DEBUG { 18:39:06 DEBUG "metric": { 18:39:06 DEBUG "pod": "ds-idrepo-2" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG [...] 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG "pod": "ds-idrepo-0" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 1713289146, 18:39:06 DEBUG "0" 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG } 18:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289146" 18:39:06 INFO [http_cmd]: http status code OK 18:39:06 DEBUG --- status code --- 18:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:06 DEBUG --- http response --- 18:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:06 DEBUG ----- output ----- 18:39:06 DEBUG { 18:39:06 DEBUG "status": "success", 18:39:06 DEBUG "data": { 18:39:06 DEBUG "resultType": "vector", 18:39:06 DEBUG "result": [ 18:39:06 DEBUG { 18:39:06 DEBUG "metric": { 18:39:06 DEBUG "instance": "10.142.0.107:9100" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG [...] 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG "instance": "10.142.0.43:9100" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 1713289146, 18:39:06 DEBUG "0.00026666666666651885" 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG } 18:39:06 INFO [http_cmd]: http status code OK 18:39:06 DEBUG --- status code --- 18:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:06 DEBUG --- http response --- 18:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:06 DEBUG ----- output ----- 18:39:06 DEBUG { 18:39:06 DEBUG "status": "success", 18:39:06 DEBUG "data": { 18:39:06 DEBUG "resultType": "vector", 18:39:06 DEBUG "result": [ 18:39:06 DEBUG { 18:39:06 DEBUG "metric": { 18:39:06 DEBUG "pod": "ds-cts-1" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG [...] 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 1713289146, 18:39:06 DEBUG "0" 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG } 18:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713289146" 18:39:06 INFO [http_cmd]: http status code OK 18:39:06 DEBUG --- status code --- 18:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:06 DEBUG --- http response --- 18:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:39:06 DEBUG ----- output ----- 18:39:06 DEBUG { 18:39:06 DEBUG "status": "success", 18:39:06 DEBUG "data": { 18:39:06 DEBUG "resultType": "vector", 18:39:06 DEBUG "result": [ 18:39:06 DEBUG { 18:39:06 DEBUG "metric": { 18:39:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG [...] 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 1713289146, 18:39:06 DEBUG "0.005" 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG } 18:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289146" 18:39:06 INFO [http_cmd]: http status code OK 18:39:06 DEBUG --- status code --- 18:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:06 DEBUG --- http response --- 18:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:06 DEBUG ----- output ----- 18:39:06 DEBUG { 18:39:06 DEBUG "status": "success", 18:39:06 DEBUG "data": { 18:39:06 DEBUG "resultType": "vector", 18:39:06 DEBUG "result": [ 18:39:06 DEBUG { 18:39:06 DEBUG "metric": { 18:39:06 DEBUG "pod": "ds-cts-1" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG [...] 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 1713289146, 18:39:06 DEBUG "0" 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG } 18:39:06 INFO 18:39:06 INFO [loop_until]: kubectl --namespace=xlou top pods 18:39:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:39:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289146" 18:39:06 INFO [http_cmd]: http status code OK 18:39:06 DEBUG --- status code --- 18:39:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:39:06 DEBUG --- http response --- 18:39:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:39:06 DEBUG ----- output ----- 18:39:06 DEBUG { 18:39:06 DEBUG "status": "success", 18:39:06 DEBUG "data": { 18:39:06 DEBUG "resultType": "vector", 18:39:06 DEBUG "result": [ 18:39:06 DEBUG { 18:39:06 DEBUG "metric": { 18:39:06 DEBUG "pod": "ds-cts-1" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG [...] 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG 18:39:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:39:06 DEBUG }, 18:39:06 DEBUG "value": [ 18:39:06 DEBUG 1713289146, 18:39:06 DEBUG "0" 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG ] 18:39:06 DEBUG } 18:39:06 DEBUG } 18:39:06 INFO [loop_until]: OK (rc = 0) 18:39:06 DEBUG --- stdout --- 18:39:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 5m 3247Mi am-555f4db595-fz8sn 4m 3221Mi am-555f4db595-kzxsp 12m 1127Mi ds-cts-0 21m 8334Mi ds-cts-1 5m 8305Mi ds-cts-2 4m 8446Mi ds-idrepo-0 10m 5042Mi ds-idrepo-1 8m 2779Mi ds-idrepo-2 7m 2858Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1041Mi idm-b85f9d978-rl76f 4m 1225Mi lodemon-69f8bd86c8-8fxnk 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 1497Mi 18:39:06 DEBUG --- stderr --- 18:39:06 DEBUG 18:39:06 INFO 18:39:06 INFO [loop_until]: kubectl --namespace=xlou top node 18:39:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:39:06 INFO [loop_until]: OK (rc = 0) 18:39:06 DEBUG --- stdout --- 18:39:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 5086Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 67m 0% 10819Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 5432Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 104m 0% 2685Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 67m 0% 6151Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 13015Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 73m 0% 2981Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 55m 0% 5018Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 133m 0% 5540Mi 4% gke-xlou-cdm-ds-0f601751-j488 56m 0% 10348Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 123m 0% 3240Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 87m 0% 2965Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 62m 0% 7324Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 75m 0% 5400Mi 9% 18:39:06 DEBUG --- stderr --- 18:39:06 DEBUG 18:40:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289205" 18:40:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289205" 18:40:05 INFO [http_cmd]: http status code OK 18:40:05 DEBUG --- status code --- 18:40:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:05 DEBUG --- http response --- 18:40:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:40:05 INFO [http_cmd]: http status code OK 18:40:05 DEBUG --- status code --- 18:40:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:05 DEBUG --- http response --- 18:40:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:40:05 DEBUG ----- output ----- 18:40:05 DEBUG { 18:40:05 DEBUG "status": "success", 18:40:05 DEBUG "data": { 18:40:05 DEBUG "resultType": "vector", 18:40:05 DEBUG "result": [ 18:40:05 DEBUG { 18:40:05 DEBUG "metric": { 18:40:05 DEBUG "pod": "ds-idrepo-2" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG [...] 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG "pod": "ds-idrepo-0" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 1713289205, 18:40:05 DEBUG "0" 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG } 18:40:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713289205" 18:40:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713289205" 18:40:05 INFO [http_cmd]: http status code OK 18:40:05 DEBUG --- status code --- 18:40:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:05 DEBUG --- http response --- 18:40:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:05 DEBUG ----- output ----- 18:40:05 DEBUG { 18:40:05 DEBUG "status": "success", 18:40:05 DEBUG "data": { 18:40:05 DEBUG "resultType": "vector", 18:40:05 DEBUG "result": [ 18:40:05 DEBUG { 18:40:05 DEBUG "metric": { 18:40:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG [...] 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 1713289205, 18:40:05 DEBUG "1668009984" 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG } 18:40:05 INFO [http_cmd]: http status code OK 18:40:05 DEBUG --- status code --- 18:40:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:05 DEBUG --- http response --- 18:40:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:40:05 DEBUG ----- output ----- 18:40:05 DEBUG { 18:40:05 DEBUG "status": "success", 18:40:05 DEBUG "data": { 18:40:05 DEBUG "resultType": "vector", 18:40:05 DEBUG "result": [ 18:40:05 DEBUG { 18:40:05 DEBUG "metric": { 18:40:05 DEBUG "pod": "ds-idrepo-2" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG [...] 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG "pod": "ds-cts-2" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 1713289205, 18:40:05 DEBUG "0" 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG } 18:40:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289205" 18:40:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289205" 18:40:05 INFO [http_cmd]: http status code OK 18:40:05 DEBUG --- status code --- 18:40:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:05 DEBUG --- http response --- 18:40:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:40:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289205" 18:40:05 INFO [http_cmd]: http status code OK 18:40:05 DEBUG --- status code --- 18:40:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:05 DEBUG --- http response --- 18:40:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:05 DEBUG ----- output ----- 18:40:05 DEBUG { 18:40:05 DEBUG "status": "success", 18:40:05 DEBUG "data": { 18:40:05 DEBUG "resultType": "vector", 18:40:05 DEBUG "result": [ 18:40:05 DEBUG { 18:40:05 DEBUG "metric": { 18:40:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG [...] 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 1713289205, 18:40:05 DEBUG "0" 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG } 18:40:05 INFO [http_cmd]: http status code OK 18:40:05 DEBUG --- status code --- 18:40:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:05 DEBUG --- http response --- 18:40:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:40:05 DEBUG ----- output ----- 18:40:05 DEBUG { 18:40:05 DEBUG "status": "success", 18:40:05 DEBUG "data": { 18:40:05 DEBUG "resultType": "vector", 18:40:05 DEBUG "result": [ 18:40:05 DEBUG { 18:40:05 DEBUG "metric": { 18:40:05 DEBUG "pod": "ds-idrepo-2" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG [...] 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG "pod": "ds-idrepo-0" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 1713289205, 18:40:05 DEBUG "0" 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG } 18:40:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289205" 18:40:05 INFO [http_cmd]: http status code OK 18:40:05 DEBUG --- status code --- 18:40:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:05 DEBUG --- http response --- 18:40:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:05 DEBUG ----- output ----- 18:40:05 DEBUG { 18:40:05 DEBUG "status": "success", 18:40:05 DEBUG "data": { 18:40:05 DEBUG "resultType": "vector", 18:40:05 DEBUG "result": [ 18:40:05 DEBUG { 18:40:05 DEBUG "metric": { 18:40:05 DEBUG "instance": "10.142.0.107:9100" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG [...] 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG "instance": "10.142.0.43:9100" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 1713289205, 18:40:05 DEBUG "9431.366666666667" 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG } 18:40:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713289205" 18:40:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289205" 18:40:05 ERROR [http_cmd]: ERROR 18:40:05 ERROR --- status code --- 18:40:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:40:05 ERROR --- http response --- 18:40:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:40:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:40:05 INFO [http_cmd]: http status code OK 18:40:05 DEBUG --- status code --- 18:40:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:05 DEBUG --- http response --- 18:40:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:05 DEBUG ----- output ----- 18:40:05 DEBUG { 18:40:05 DEBUG "status": "success", 18:40:05 DEBUG "data": { 18:40:05 DEBUG "resultType": "vector", 18:40:05 DEBUG "result": [ 18:40:05 DEBUG { 18:40:05 DEBUG "metric": { 18:40:05 DEBUG "pod": "ds-idrepo-2" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG [...] 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG "pod": "ds-idrepo-0" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 1713289205, 18:40:05 DEBUG "0" 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG } 18:40:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289205" 18:40:05 INFO [http_cmd]: http status code OK 18:40:05 DEBUG --- status code --- 18:40:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:05 DEBUG --- http response --- 18:40:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:40:05 DEBUG ----- output ----- 18:40:05 DEBUG { 18:40:05 DEBUG "status": "success", 18:40:05 DEBUG "data": { 18:40:05 DEBUG "resultType": "vector", 18:40:05 DEBUG "result": [ 18:40:05 DEBUG { 18:40:05 DEBUG "metric": { 18:40:05 DEBUG "pod": "ds-idrepo-2" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG [...] 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG 18:40:05 DEBUG "pod": "ds-idrepo-0" 18:40:05 DEBUG }, 18:40:05 DEBUG "value": [ 18:40:05 DEBUG 1713289205, 18:40:05 DEBUG "0" 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG ] 18:40:05 DEBUG } 18:40:05 DEBUG } 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289206" 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289206" 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289206" 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "0" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "instance": "10.142.0.107:9100" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "instance": "10.142.0.43:9100" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "19836851200" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "instance": "10.142.0.107:9100" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "instance": "10.142.0.43:9100" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "0.00033333333333333365" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289206" 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713289206" 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "instance": "10.142.0.107:9100" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "instance": "10.142.0.43:9100" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "16364.066666666664" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "0.027030349492595057" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289206" 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289206" 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "2" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "pod": "ds-idrepo-2" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "pod": "ds-idrepo-0" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "0" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289206" 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289206" 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "0" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "0" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289206" 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289206" 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "pod": "ds-cts-1" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "pod": "ds-cts-2" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "0" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "pod": "ds-cts-1" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "pod": "ds-cts-2" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "0" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289206" 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713289206" 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "0.005" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "pod": "ds-idrepo-2" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "pod": "ds-idrepo-0" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "0" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289206" 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "instance": "10.142.0.107:9100" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "instance": "10.142.0.43:9100" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "0.0008000000000000303" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289206" 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "pod": "ds-cts-1" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "0" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO 18:40:06 INFO [loop_until]: kubectl --namespace=xlou top pods 18:40:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:40:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289206" 18:40:06 INFO [http_cmd]: http status code OK 18:40:06 DEBUG --- status code --- 18:40:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:40:06 DEBUG --- http response --- 18:40:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 18:40:06 DEBUG ----- output ----- 18:40:06 DEBUG { 18:40:06 DEBUG "status": "success", 18:40:06 DEBUG "data": { 18:40:06 DEBUG "resultType": "vector", 18:40:06 DEBUG "result": [ 18:40:06 DEBUG { 18:40:06 DEBUG "metric": { 18:40:06 DEBUG "pod": "ds-cts-1" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG [...] 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG 18:40:06 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 18:40:06 DEBUG }, 18:40:06 DEBUG "value": [ 18:40:06 DEBUG 1713289206, 18:40:06 DEBUG "0" 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG ] 18:40:06 DEBUG } 18:40:06 DEBUG } 18:40:06 INFO [loop_until]: OK (rc = 0) 18:40:06 DEBUG --- stdout --- 18:40:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 11m 3242Mi am-555f4db595-fz8sn 5m 3221Mi am-555f4db595-kzxsp 6m 1127Mi ds-cts-0 5m 8290Mi ds-cts-1 5m 8306Mi ds-cts-2 5m 8446Mi ds-idrepo-0 483m 5029Mi ds-idrepo-1 290m 2780Mi ds-idrepo-2 420m 2889Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1043Mi idm-b85f9d978-rl76f 3m 1226Mi lodemon-69f8bd86c8-8fxnk 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 1497Mi 18:40:06 DEBUG --- stderr --- 18:40:06 DEBUG 18:40:06 INFO 18:40:06 INFO [loop_until]: kubectl --namespace=xlou top node 18:40:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:40:06 INFO [loop_until]: OK (rc = 0) 18:40:06 DEBUG --- stdout --- 18:40:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5084Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 65m 0% 10800Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 5435Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 2679Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 440m 1% 6152Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 149m 0% 13000Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2979Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 63m 0% 5011Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 334m 1% 5415Mi 4% gke-xlou-cdm-ds-0f601751-j488 56m 0% 10328Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 80m 0% 3227Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 88m 0% 2979Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 707m 2% 7299Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 76m 0% 5400Mi 9% 18:40:06 DEBUG --- stderr --- 18:40:06 DEBUG 18:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289265" 18:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289265" 18:41:05 INFO [http_cmd]: http status code OK 18:41:05 DEBUG --- status code --- 18:41:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:05 DEBUG --- http response --- 18:41:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713289265" 18:41:05 INFO [http_cmd]: http status code OK 18:41:05 DEBUG --- status code --- 18:41:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:05 DEBUG --- http response --- 18:41:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:41:05 DEBUG ----- output ----- 18:41:05 DEBUG { 18:41:05 DEBUG "status": "success", 18:41:05 DEBUG "data": { 18:41:05 DEBUG "resultType": "vector", 18:41:05 DEBUG "result": [ 18:41:05 DEBUG { 18:41:05 DEBUG "metric": { 18:41:05 DEBUG "pod": "ds-idrepo-2" 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG [...] 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG "pod": "ds-idrepo-0" 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 1713289265, 18:41:05 DEBUG "0" 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG } 18:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289265" 18:41:05 INFO [http_cmd]: http status code OK 18:41:05 DEBUG --- status code --- 18:41:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:05 DEBUG --- http response --- 18:41:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:41:05 DEBUG ----- output ----- 18:41:05 DEBUG { 18:41:05 DEBUG "status": "success", 18:41:05 DEBUG "data": { 18:41:05 DEBUG "resultType": "vector", 18:41:05 DEBUG "result": [ 18:41:05 DEBUG { 18:41:05 DEBUG "metric": { 18:41:05 DEBUG "pod": "ds-idrepo-2" 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG [...] 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG "pod": "ds-cts-2" 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 1713289265, 18:41:05 DEBUG "0" 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG } 18:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713289265" 18:41:05 INFO [http_cmd]: http status code OK 18:41:05 DEBUG --- status code --- 18:41:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:05 DEBUG --- http response --- 18:41:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:05 DEBUG ----- output ----- 18:41:05 DEBUG { 18:41:05 DEBUG "status": "success", 18:41:05 DEBUG "data": { 18:41:05 DEBUG "resultType": "vector", 18:41:05 DEBUG "result": [ 18:41:05 DEBUG { 18:41:05 DEBUG "metric": { 18:41:05 DEBUG "instance": "10.142.0.107:9100" 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG [...] 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG "instance": "10.142.0.43:9100" 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 1713289265, 18:41:05 DEBUG "10440.066666666668" 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG } 18:41:05 INFO [http_cmd]: http status code OK 18:41:05 DEBUG --- status code --- 18:41:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:05 DEBUG --- http response --- 18:41:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:05 DEBUG ----- output ----- 18:41:05 DEBUG { 18:41:05 DEBUG "status": "success", 18:41:05 DEBUG "data": { 18:41:05 DEBUG "resultType": "vector", 18:41:05 DEBUG "result": [ 18:41:05 DEBUG { 18:41:05 DEBUG "metric": { 18:41:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG [...] 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 1713289265, 18:41:05 DEBUG "1665429504" 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG } 18:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289265" 18:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289265" 18:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289265" 18:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289265" 18:41:05 INFO [http_cmd]: http status code OK 18:41:05 DEBUG --- status code --- 18:41:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:05 DEBUG --- http response --- 18:41:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:41:05 INFO [http_cmd]: http status code OK 18:41:05 DEBUG --- status code --- 18:41:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:05 DEBUG --- http response --- 18:41:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:05 DEBUG ----- output ----- 18:41:05 DEBUG { 18:41:05 DEBUG "status": "success", 18:41:05 DEBUG "data": { 18:41:05 DEBUG "resultType": "vector", 18:41:05 DEBUG "result": [ 18:41:05 DEBUG { 18:41:05 DEBUG "metric": { 18:41:05 DEBUG "pod": "ds-idrepo-2" 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG [...] 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG "pod": "ds-idrepo-0" 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 1713289265, 18:41:05 DEBUG "0" 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG } 18:41:05 INFO [http_cmd]: http status code OK 18:41:05 INFO [http_cmd]: http status code OK 18:41:05 DEBUG --- status code --- 18:41:05 DEBUG --- status code --- 18:41:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:05 DEBUG --- http response --- 18:41:05 DEBUG --- http response --- 18:41:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:41:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:05 DEBUG ----- output ----- 18:41:05 DEBUG ----- output ----- 18:41:05 DEBUG { 18:41:05 DEBUG { 18:41:05 DEBUG "status": "success", 18:41:05 DEBUG "status": "success", 18:41:05 DEBUG "data": { 18:41:05 DEBUG "data": { 18:41:05 DEBUG "resultType": "vector", 18:41:05 DEBUG "resultType": "vector", 18:41:05 DEBUG "result": [ 18:41:05 DEBUG "result": [ 18:41:05 DEBUG { 18:41:05 DEBUG { 18:41:05 DEBUG "metric": { 18:41:05 DEBUG "metric": { 18:41:05 DEBUG "pod": "ds-idrepo-2" 18:41:05 DEBUG "pod": "am-555f4db595-fz8sn" 18:41:05 DEBUG }, 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG [...] 18:41:05 DEBUG [...] 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG "pod": "ds-idrepo-0" 18:41:05 DEBUG "pod": "am-555f4db595-kzxsp" 18:41:05 DEBUG }, 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 1713289265, 18:41:05 DEBUG 1713289265, 18:41:05 DEBUG "0" 18:41:05 DEBUG "0" 18:41:05 DEBUG ] 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG } 18:41:05 DEBUG ] 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG } 18:41:05 DEBUG } 18:41:05 DEBUG } 18:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713289265" 18:41:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289265" 18:41:05 ERROR [http_cmd]: ERROR 18:41:05 ERROR --- status code --- 18:41:05 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:41:05 ERROR --- http response --- 18:41:05 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:41:05 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:41:05 INFO [http_cmd]: http status code OK 18:41:05 DEBUG --- status code --- 18:41:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:05 DEBUG --- http response --- 18:41:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:41:05 DEBUG ----- output ----- 18:41:05 DEBUG { 18:41:05 DEBUG "status": "success", 18:41:05 DEBUG "data": { 18:41:05 DEBUG "resultType": "vector", 18:41:05 DEBUG "result": [ 18:41:05 DEBUG { 18:41:05 DEBUG "metric": { 18:41:05 DEBUG "pod": "ds-idrepo-2" 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG [...] 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG 18:41:05 DEBUG "pod": "ds-idrepo-0" 18:41:05 DEBUG }, 18:41:05 DEBUG "value": [ 18:41:05 DEBUG 1713289265, 18:41:05 DEBUG "0" 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG ] 18:41:05 DEBUG } 18:41:05 DEBUG } 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289266" 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "0" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289266" 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289266" 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "instance": "10.142.0.107:9100" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "instance": "10.142.0.43:9100" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "19837621248" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "instance": "10.142.0.107:9100" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "instance": "10.142.0.43:9100" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "0.0003333333333333262" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713289266" 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289266" 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "instance": "10.142.0.107:9100" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "instance": "10.142.0.43:9100" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "16117.799999999997" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "0.026074150486494217" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289266" 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289266" 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289266" 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "pod": "ds-idrepo-2" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "pod": "ds-idrepo-0" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "0" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "0" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "2" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289266" 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "pod": "ds-cts-1" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "0.000021179632496862104" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289266" 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289266" 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713289266" 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "0" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289266" 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:41:06 DEBUG "pod": "ds-idrepo-2" 18:41:06 DEBUG }, 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:41:06 DEBUG "pod": "ds-idrepo-0" 18:41:06 DEBUG }, 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "0.005" 18:41:06 DEBUG "0" 18:41:06 DEBUG ] 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289266" 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "instance": "10.142.0.107:9100" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "instance": "10.142.0.43:9100" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "0.00036666666666652263" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "pod": "ds-cts-1" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "119.65762028570593" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289266" 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "pod": "ds-cts-1" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "0" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289266" 18:41:06 INFO 18:41:06 INFO [loop_until]: kubectl --namespace=xlou top pods 18:41:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:41:06 INFO [http_cmd]: http status code OK 18:41:06 DEBUG --- status code --- 18:41:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:41:06 DEBUG --- http response --- 18:41:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 18:41:06 DEBUG ----- output ----- 18:41:06 DEBUG { 18:41:06 DEBUG "status": "success", 18:41:06 DEBUG "data": { 18:41:06 DEBUG "resultType": "vector", 18:41:06 DEBUG "result": [ 18:41:06 DEBUG { 18:41:06 DEBUG "metric": { 18:41:06 DEBUG "pod": "ds-cts-1" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG [...] 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG 18:41:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:41:06 DEBUG }, 18:41:06 DEBUG "value": [ 18:41:06 DEBUG 1713289266, 18:41:06 DEBUG "0" 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG ] 18:41:06 DEBUG } 18:41:06 DEBUG } 18:41:06 INFO [loop_until]: OK (rc = 0) 18:41:06 DEBUG --- stdout --- 18:41:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 7m 3243Mi am-555f4db595-fz8sn 4m 3223Mi am-555f4db595-kzxsp 4m 1126Mi ds-cts-0 5m 8290Mi ds-cts-1 6m 8306Mi ds-cts-2 4m 8446Mi ds-idrepo-0 8m 5030Mi ds-idrepo-1 8m 2779Mi ds-idrepo-2 8m 2857Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1042Mi idm-b85f9d978-rl76f 4m 1226Mi lodemon-69f8bd86c8-8fxnk 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 1497Mi 18:41:06 DEBUG --- stderr --- 18:41:06 DEBUG 18:41:06 INFO 18:41:06 INFO [loop_until]: kubectl --namespace=xlou top node 18:41:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:41:06 INFO [loop_until]: OK (rc = 0) 18:41:06 DEBUG --- stdout --- 18:41:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5085Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 67m 0% 10801Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 5442Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 2674Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 63m 0% 6139Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 129m 0% 13006Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2981Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 59m 0% 5004Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5433Mi 4% gke-xlou-cdm-ds-0f601751-j488 55m 0% 10338Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 81m 0% 3225Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 83m 0% 2993Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 60m 0% 7317Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 71m 0% 5403Mi 9% 18:41:06 DEBUG --- stderr --- 18:41:06 DEBUG 18:42:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289325" 18:42:05 INFO [http_cmd]: http status code OK 18:42:05 DEBUG --- status code --- 18:42:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:05 DEBUG --- http response --- 18:42:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:42:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289325" 18:42:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289325" 18:42:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713289325" 18:42:05 INFO [http_cmd]: http status code OK 18:42:05 DEBUG --- status code --- 18:42:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:05 DEBUG --- http response --- 18:42:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:42:05 DEBUG ----- output ----- 18:42:05 DEBUG { 18:42:05 DEBUG "status": "success", 18:42:05 DEBUG "data": { 18:42:05 DEBUG "resultType": "vector", 18:42:05 DEBUG "result": [ 18:42:05 DEBUG { 18:42:05 DEBUG "metric": { 18:42:05 DEBUG "pod": "ds-idrepo-2" 18:42:05 DEBUG }, 18:42:05 DEBUG "value": [ 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG [...] 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG "pod": "ds-idrepo-0" 18:42:05 DEBUG }, 18:42:05 DEBUG "value": [ 18:42:05 DEBUG 1713289325, 18:42:05 DEBUG "0" 18:42:05 DEBUG ] 18:42:05 DEBUG } 18:42:05 DEBUG ] 18:42:05 DEBUG } 18:42:05 DEBUG } 18:42:05 INFO [http_cmd]: http status code OK 18:42:05 DEBUG --- status code --- 18:42:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:05 DEBUG --- http response --- 18:42:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:42:05 DEBUG ----- output ----- 18:42:05 DEBUG { 18:42:05 DEBUG "status": "success", 18:42:05 DEBUG "data": { 18:42:05 DEBUG "resultType": "vector", 18:42:05 DEBUG "result": [ 18:42:05 DEBUG { 18:42:05 DEBUG "metric": { 18:42:05 DEBUG "pod": "ds-idrepo-2" 18:42:05 DEBUG }, 18:42:05 DEBUG "value": [ 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG [...] 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG "pod": "ds-idrepo-0" 18:42:05 DEBUG }, 18:42:05 DEBUG "value": [ 18:42:05 DEBUG 1713289325, 18:42:05 DEBUG "0" 18:42:05 DEBUG ] 18:42:05 DEBUG } 18:42:05 DEBUG ] 18:42:05 DEBUG } 18:42:05 DEBUG } 18:42:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289325" 18:42:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289325" 18:42:05 INFO [http_cmd]: http status code OK 18:42:05 DEBUG --- status code --- 18:42:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:05 DEBUG --- http response --- 18:42:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:42:05 DEBUG ----- output ----- 18:42:05 DEBUG { 18:42:05 DEBUG "status": "success", 18:42:05 DEBUG "data": { 18:42:05 DEBUG "resultType": "vector", 18:42:05 DEBUG "result": [ 18:42:05 DEBUG { 18:42:05 DEBUG "metric": { 18:42:05 DEBUG "pod": "ds-idrepo-2" 18:42:05 DEBUG }, 18:42:05 DEBUG "value": [ 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG [...] 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG "pod": "ds-cts-2" 18:42:05 DEBUG }, 18:42:05 DEBUG "value": [ 18:42:05 DEBUG 1713289325, 18:42:05 DEBUG "0" 18:42:05 DEBUG ] 18:42:05 DEBUG } 18:42:05 DEBUG ] 18:42:05 DEBUG } 18:42:05 DEBUG } 18:42:05 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713289325" 18:42:05 INFO [http_cmd]: http status code OK 18:42:05 DEBUG --- status code --- 18:42:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:05 DEBUG --- http response --- 18:42:05 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:42:05 INFO [http_cmd]: http status code OK 18:42:05 DEBUG --- status code --- 18:42:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:05 DEBUG --- http response --- 18:42:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:05 DEBUG ----- output ----- 18:42:05 DEBUG { 18:42:05 DEBUG "status": "success", 18:42:05 DEBUG "data": { 18:42:05 DEBUG "resultType": "vector", 18:42:05 DEBUG "result": [ 18:42:05 DEBUG { 18:42:05 DEBUG "metric": { 18:42:05 DEBUG "instance": "10.142.0.107:9100" 18:42:05 DEBUG }, 18:42:05 DEBUG "value": [ 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG [...] 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG "instance": "10.142.0.43:9100" 18:42:05 DEBUG }, 18:42:05 DEBUG "value": [ 18:42:05 DEBUG 1713289325, 18:42:05 DEBUG "9008.8" 18:42:05 DEBUG ] 18:42:05 DEBUG } 18:42:05 DEBUG ] 18:42:05 DEBUG } 18:42:05 DEBUG } 18:42:05 INFO [http_cmd]: http status code OK 18:42:05 DEBUG --- status code --- 18:42:05 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:05 DEBUG --- http response --- 18:42:05 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:05 DEBUG ----- output ----- 18:42:05 DEBUG { 18:42:05 DEBUG "status": "success", 18:42:05 DEBUG "data": { 18:42:05 DEBUG "resultType": "vector", 18:42:05 DEBUG "result": [ 18:42:05 DEBUG { 18:42:05 DEBUG "metric": { 18:42:05 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:42:05 DEBUG }, 18:42:05 DEBUG "value": [ 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG [...] 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG 18:42:05 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:42:05 DEBUG }, 18:42:05 DEBUG "value": [ 18:42:05 DEBUG 1713289325, 18:42:05 DEBUG "1665249280" 18:42:05 DEBUG ] 18:42:05 DEBUG } 18:42:05 DEBUG ] 18:42:05 DEBUG } 18:42:05 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713289326" 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 ERROR [http_cmd]: ERROR 18:42:06 ERROR --- status code --- 18:42:06 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:42:06 ERROR --- http response --- 18:42:06 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:42:06 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "ds-idrepo-2" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "ds-idrepo-0" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "ds-idrepo-2" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "ds-idrepo-0" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "instance": "10.142.0.107:9100" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "instance": "10.142.0.43:9100" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "16142.999999999998" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713289326" 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289326" 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0.021494724818349433" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "instance": "10.142.0.107:9100" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "instance": "10.142.0.43:9100" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "19838311424" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "ds-idrepo-2" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "ds-idrepo-0" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "instance": "10.142.0.107:9100" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "instance": "10.142.0.43:9100" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0.0003333333333333262" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "ds-cts-1" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0.00002376760563381719" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "ds-idrepo-2" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "ds-idrepo-0" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "2" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0.005" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "am-555f4db595-fz8sn" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "am-555f4db595-kzxsp" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "instance": "10.142.0.107:9100" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "instance": "10.142.0.43:9100" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0.000599999999999549" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "ds-cts-1" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "ds-cts-1" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713289326" 18:42:06 INFO [http_cmd]: http status code OK 18:42:06 DEBUG --- status code --- 18:42:06 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:42:06 DEBUG --- http response --- 18:42:06 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:42:06 DEBUG ----- output ----- 18:42:06 DEBUG { 18:42:06 DEBUG "status": "success", 18:42:06 DEBUG "data": { 18:42:06 DEBUG "resultType": "vector", 18:42:06 DEBUG "result": [ 18:42:06 DEBUG { 18:42:06 DEBUG "metric": { 18:42:06 DEBUG "pod": "ds-cts-1" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG [...] 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG 18:42:06 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:42:06 DEBUG }, 18:42:06 DEBUG "value": [ 18:42:06 DEBUG 1713289326, 18:42:06 DEBUG "0" 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG ] 18:42:06 DEBUG } 18:42:06 DEBUG } 18:42:06 INFO 18:42:06 INFO [loop_until]: kubectl --namespace=xlou top pods 18:42:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:42:06 INFO [loop_until]: OK (rc = 0) 18:42:06 DEBUG --- stdout --- 18:42:06 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 9Mi am-555f4db595-4lbss 3m 3242Mi am-555f4db595-fz8sn 7m 3223Mi am-555f4db595-kzxsp 4m 1126Mi ds-cts-0 5m 8290Mi ds-cts-1 5m 8306Mi ds-cts-2 4m 8446Mi ds-idrepo-0 8m 5029Mi ds-idrepo-1 8m 2780Mi ds-idrepo-2 8m 2856Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1042Mi idm-b85f9d978-rl76f 4m 1226Mi lodemon-69f8bd86c8-8fxnk 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 1497Mi 18:42:06 DEBUG --- stderr --- 18:42:06 DEBUG 18:42:06 INFO 18:42:06 INFO [loop_until]: kubectl --namespace=xlou top node 18:42:06 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:42:06 INFO [loop_until]: OK (rc = 0) 18:42:06 DEBUG --- stdout --- 18:42:06 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5076Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 10814Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 5441Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 2675Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 59m 0% 6150Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 13005Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2977Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 57m 0% 4995Mi 4% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5437Mi 4% gke-xlou-cdm-ds-0f601751-j488 58m 0% 10338Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 80m 0% 3221Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 89m 0% 2999Mi 2% gke-xlou-cdm-ds-0f601751-nmxm 60m 0% 7329Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 77m 0% 5403Mi 9% 18:42:06 DEBUG --- stderr --- 18:42:06 DEBUG 18:42:59 INFO Finished: True 18:42:59 INFO Waiting for threads to register finish flag 18:43:06 INFO Done. Have a nice day! :) 127.0.0.1 - - [16/Apr/2024 18:43:06] "GET /monitoring/stop HTTP/1.1" 200 - 18:43:10 WARNING File /tmp/lodemon_data-20240416_182702/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p5373', ...) WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6471', ...) 127.0.0.1 - - [16/Apr/2024 18:43:14] "GET /monitoring/process HTTP/1.1" 200 -