==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-7cfb6bd874-prp2f Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Wed, 17 Apr 2024 21:42:20 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=7cfb6bd874 skaffold.dev/run-id=c3f8c81f-12a6-4de1-8dd9-a74c1179d62e Annotations: Status: Running IP: 10.106.46.164 IPs: IP: 10.106.46.164 Controlled By: ReplicaSet/lodemon-7cfb6bd874 Containers: lodemon: Container ID: containerd://c4d9b90eaa83a2851d042318d2b25ded9d6b79a406db8d07134a2a0121ec0f60 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:2febaeab396557db3e0fc07c0501fa786a35d3473828be1a84a3d1610beec5de Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Wed, 17 Apr 2024 21:42:21 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5ng5v (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-5ng5v: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 42m default-scheduler Successfully assigned xlou/lodemon-7cfb6bd874-prp2f to gke-xlou-cdm-ds-0f601751-mc5d Normal Pulling 42m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 42m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 371.308762ms (371.315569ms including waiting) Normal Created 42m kubelet Created container lodemon Normal Started 42m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.46.164:8080 Press CTRL+C to quit 22:43:27 INFO 22:43:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:27 INFO [loop_until]: OK (rc = 0) 22:43:27 DEBUG --- stdout --- 22:43:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:27 DEBUG --- stderr --- 22:43:27 DEBUG 22:43:27 INFO 22:43:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:27 INFO [loop_until]: OK (rc = 0) 22:43:27 DEBUG --- stdout --- 22:43:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:27 DEBUG --- stderr --- 22:43:27 DEBUG 22:43:27 INFO 22:43:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:27 INFO [loop_until]: OK (rc = 0) 22:43:27 DEBUG --- stdout --- 22:43:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:27 DEBUG --- stderr --- 22:43:27 DEBUG 22:43:28 INFO 22:43:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:28 INFO [loop_until]: OK (rc = 0) 22:43:28 DEBUG --- stdout --- 22:43:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:28 DEBUG --- stderr --- 22:43:28 DEBUG 22:43:28 INFO 22:43:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:28 INFO [loop_until]: OK (rc = 0) 22:43:28 DEBUG --- stdout --- 22:43:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:28 DEBUG --- stderr --- 22:43:28 DEBUG 22:43:28 INFO 22:43:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:28 INFO [loop_until]: OK (rc = 0) 22:43:28 DEBUG --- stdout --- 22:43:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:28 DEBUG --- stderr --- 22:43:28 DEBUG 22:43:28 INFO 22:43:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:28 INFO [loop_until]: OK (rc = 0) 22:43:28 DEBUG --- stdout --- 22:43:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:28 DEBUG --- stderr --- 22:43:28 DEBUG 22:43:28 INFO 22:43:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:28 INFO [loop_until]: OK (rc = 0) 22:43:28 DEBUG --- stdout --- 22:43:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:28 DEBUG --- stderr --- 22:43:28 DEBUG 22:43:28 INFO 22:43:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:28 INFO [loop_until]: OK (rc = 0) 22:43:28 DEBUG --- stdout --- 22:43:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:28 DEBUG --- stderr --- 22:43:28 DEBUG 22:43:28 INFO 22:43:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:29 INFO [loop_until]: OK (rc = 0) 22:43:29 DEBUG --- stdout --- 22:43:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:29 DEBUG --- stderr --- 22:43:29 DEBUG 22:43:29 INFO 22:43:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:29 INFO [loop_until]: OK (rc = 0) 22:43:29 DEBUG --- stdout --- 22:43:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:29 DEBUG --- stderr --- 22:43:29 DEBUG 22:43:29 INFO 22:43:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:29 INFO [loop_until]: OK (rc = 0) 22:43:29 DEBUG --- stdout --- 22:43:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:29 DEBUG --- stderr --- 22:43:29 DEBUG 22:43:29 INFO 22:43:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:29 INFO [loop_until]: OK (rc = 0) 22:43:29 DEBUG --- stdout --- 22:43:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:29 DEBUG --- stderr --- 22:43:29 DEBUG 22:43:29 INFO 22:43:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:29 INFO [loop_until]: OK (rc = 0) 22:43:29 DEBUG --- stdout --- 22:43:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:29 DEBUG --- stderr --- 22:43:29 DEBUG 22:43:29 INFO 22:43:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:29 INFO [loop_until]: OK (rc = 0) 22:43:29 DEBUG --- stdout --- 22:43:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:29 DEBUG --- stderr --- 22:43:29 DEBUG 22:43:29 INFO 22:43:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:29 INFO [loop_until]: OK (rc = 0) 22:43:29 DEBUG --- stdout --- 22:43:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:29 DEBUG --- stderr --- 22:43:29 DEBUG 22:43:29 INFO 22:43:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:29 INFO [loop_until]: OK (rc = 0) 22:43:29 DEBUG --- stdout --- 22:43:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:29 DEBUG --- stderr --- 22:43:29 DEBUG 22:43:30 INFO 22:43:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:30 INFO [loop_until]: OK (rc = 0) 22:43:30 DEBUG --- stdout --- 22:43:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:30 DEBUG --- stderr --- 22:43:30 DEBUG 22:43:30 INFO 22:43:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:30 INFO [loop_until]: OK (rc = 0) 22:43:30 DEBUG --- stdout --- 22:43:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:30 DEBUG --- stderr --- 22:43:30 DEBUG 22:43:30 INFO 22:43:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:30 INFO [loop_until]: OK (rc = 0) 22:43:30 DEBUG --- stdout --- 22:43:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:30 DEBUG --- stderr --- 22:43:30 DEBUG 22:43:30 INFO 22:43:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:30 INFO [loop_until]: OK (rc = 0) 22:43:30 DEBUG --- stdout --- 22:43:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:30 DEBUG --- stderr --- 22:43:30 DEBUG 22:43:30 INFO 22:43:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:30 INFO [loop_until]: OK (rc = 0) 22:43:30 DEBUG --- stdout --- 22:43:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:30 DEBUG --- stderr --- 22:43:30 DEBUG 22:43:30 INFO 22:43:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:30 INFO [loop_until]: OK (rc = 0) 22:43:30 DEBUG --- stdout --- 22:43:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:30 DEBUG --- stderr --- 22:43:30 DEBUG 22:43:30 INFO 22:43:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:30 INFO [loop_until]: OK (rc = 0) 22:43:30 DEBUG --- stdout --- 22:43:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:30 DEBUG --- stderr --- 22:43:30 DEBUG 22:43:31 INFO 22:43:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:31 INFO [loop_until]: OK (rc = 0) 22:43:31 DEBUG --- stdout --- 22:43:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:31 DEBUG --- stderr --- 22:43:31 DEBUG 22:43:31 INFO 22:43:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:31 INFO [loop_until]: OK (rc = 0) 22:43:31 DEBUG --- stdout --- 22:43:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:31 DEBUG --- stderr --- 22:43:31 DEBUG 22:43:31 INFO 22:43:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:43:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:31 INFO [loop_until]: OK (rc = 0) 22:43:31 DEBUG --- stdout --- 22:43:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:43:31 DEBUG --- stderr --- 22:43:31 DEBUG 22:43:31 INFO Initializing monitoring instance threads 22:43:31 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 22:43:31 INFO Starting instance threads 22:43:31 INFO 22:43:31 INFO Thread started 22:43:31 INFO [loop_until]: kubectl --namespace=xlou top node 22:43:31 INFO 22:43:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:31 INFO Thread started 22:43:31 INFO [loop_until]: kubectl --namespace=xlou top pods 22:43:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 DEBUG --- status code --- 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG --- status code --- 22:43:31 INFO Thread started 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG --- status code --- 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 DEBUG --- status code --- 22:43:31 INFO Thread started 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:43:31 DEBUG --- status code --- 22:43:31 INFO Thread started 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:31 DEBUG { 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 DEBUG ----- output ----- 22:43:31 INFO Thread started 22:43:31 INFO [loop_until]: OK (rc = 0) 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG { 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG { 22:43:31 DEBUG --- stdout --- 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 DEBUG { 22:43:31 INFO Thread started 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG { 22:43:31 DEBUG { 22:43:31 DEBUG "data": { 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG { 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG { 22:43:31 DEBUG { 22:43:31 DEBUG { 22:43:31 DEBUG { 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 7m 8430Mi am-b59c765b7-bgsxp 7m 8593Mi am-b59c765b7-ch5q4 7m 8275Mi ds-cts-0 12m 7792Mi ds-cts-1 13m 7574Mi ds-cts-2 12m 7851Mi ds-idrepo-0 16m 3142Mi ds-idrepo-1 16m 2836Mi ds-idrepo-2 16m 2766Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1010Mi lodemon-7cfb6bd874-prp2f 1m 54Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1m 1457Mi 22:43:31 DEBUG "status": "success", 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 DEBUG { 22:43:31 INFO Thread started 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG { 22:43:31 DEBUG { 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG "data": { 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG "data": { 22:43:31 DEBUG --- stderr --- 22:43:31 DEBUG "data": { 22:43:31 DEBUG "status": "success", 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 DEBUG --- http response --- 22:43:31 INFO Thread started 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG "data": { 22:43:31 DEBUG "data": { 22:43:31 DEBUG "result": [ 22:43:31 DEBUG "data": { 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG "data": { 22:43:31 DEBUG "data": { 22:43:31 DEBUG "data": { 22:43:31 DEBUG "data": { 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG { 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG "data": { 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 INFO [loop_until]: OK (rc = 0) 22:43:31 DEBUG "data": { 22:43:31 INFO Thread started 22:43:31 DEBUG "data": { 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG { 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG "result": [ 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG "result": [ 22:43:31 DEBUG "result": [ 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG --- stdout --- 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG "resultType": "vector", 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390211" 22:43:31 DEBUG "resultType": "vector", 22:43:31 INFO Thread started 22:43:31 DEBUG "result": [ 22:43:31 DEBUG "result": [ 22:43:31 DEBUG "metric": { 22:43:31 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:43:31 DEBUG { 22:43:31 DEBUG "result": [ 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG "result": [ 22:43:31 DEBUG "result": [ 22:43:31 DEBUG "result": [ 22:43:31 DEBUG { 22:43:31 DEBUG "data": { 22:43:31 DEBUG { 22:43:31 DEBUG { 22:43:31 DEBUG "result": [ 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG { 22:43:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5540Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 68m 0% 6026Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 10836Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 78m 0% 3010Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 69m 0% 9995Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 4996Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 70m 0% 11037Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 66m 0% 2878Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 67m 0% 10406Mi 9% gke-xlou-cdm-ds-0f601751-j488 65m 0% 5323Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 71m 0% 11098Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 101m 0% 5913Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 69m 0% 9524Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 66m 0% 5278Mi 9% 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG "result": [ 22:43:31 DEBUG "result": [ 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390211" 22:43:31 DEBUG { 22:43:31 INFO Thread started 22:43:31 DEBUG { 22:43:31 DEBUG "result": [ 22:43:31 DEBUG }, 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG { 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG { 22:43:31 DEBUG { 22:43:31 DEBUG { 22:43:31 DEBUG "metric": { 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG "metric": { 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG "metric": { 22:43:31 DEBUG { 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG --- stderr --- 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG { 22:43:31 DEBUG { 22:43:31 DEBUG "metric": { 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390211" 22:43:31 INFO Thread started 22:43:31 DEBUG "metric": { 22:43:31 DEBUG { 22:43:31 DEBUG "value": [ 22:43:31 DEBUG "data": { 22:43:31 DEBUG "metric": { 22:43:31 DEBUG { 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG "metric": { 22:43:31 DEBUG "metric": { 22:43:31 DEBUG "metric": { 22:43:31 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:43:31 DEBUG "result": [ 22:43:31 DEBUG "instance": "10.142.0.107:9100" 22:43:31 DEBUG --- status code --- 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:43:31 DEBUG "metric": { 22:43:31 DEBUG "data": { 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG 22:43:31 DEBUG "metric": { 22:43:31 DEBUG "metric": { 22:43:31 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:43:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390211" 22:43:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:43:31 INFO Thread started 22:43:31 DEBUG "metric": { 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "instance": "10.142.0.107:9100" 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:43:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:43:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:43:31 DEBUG }, 22:43:31 DEBUG { 22:43:31 ERROR [http_cmd]: ERROR 22:43:31 DEBUG }, 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG }, 22:43:31 DEBUG "pod": "ds-cts-0" 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:43:31 DEBUG "pod": "ds-idrepo-0" 22:43:31 DEBUG "pod": "ds-idrepo-0" 22:43:31 DEBUG }, 22:43:31 DEBUG }, 22:43:31 INFO All threads has been started 127.0.0.1 - - [17/Apr/2024 22:43:31] "GET /monitoring/start HTTP/1.1" 200 - 22:43:31 DEBUG "instance": "10.142.0.107:9100" 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG 22:43:31 DEBUG }, 22:43:31 DEBUG "data": { 22:43:31 DEBUG }, 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:31 DEBUG }, 22:43:31 DEBUG }, 22:43:31 DEBUG "value": [ 22:43:31 DEBUG "metric": { 22:43:31 ERROR --- status code --- 22:43:31 DEBUG "value": [ 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG --- http response --- 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG }, 22:43:31 DEBUG "result": [ 22:43:31 DEBUG { 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG }, 22:43:31 DEBUG }, 22:43:31 DEBUG "value": [ 22:43:31 DEBUG "value": [ 22:43:31 DEBUG }, 22:43:31 DEBUG "result": [ 22:43:31 DEBUG [...] 22:43:31 DEBUG "value": [ 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG "value": [ 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG "value": [ 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG "pod": "ds-idrepo-0" 22:43:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:43:31 DEBUG 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG "value": [ 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:43:31 DEBUG "value": [ 22:43:31 DEBUG { 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG { 22:43:31 DEBUG "value": [ 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "result": [ 22:43:31 DEBUG 22:43:31 DEBUG { 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG }, 22:43:31 ERROR --- http response --- 22:43:31 DEBUG 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG 22:43:31 DEBUG "metric": { 22:43:31 DEBUG "data": { 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG { 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG { 22:43:31 DEBUG 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG 22:43:31 DEBUG { 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG { 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG "data": { 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "metric": { 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "metric": { 22:43:31 DEBUG 22:43:31 DEBUG "data": { 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG [...] 22:43:31 DEBUG 22:43:31 DEBUG [...] 22:43:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG "data": { 22:43:31 DEBUG "data": { 22:43:31 DEBUG "resultType": "vector", 22:43:31 INFO [http_cmd]: http status code OK 22:43:31 DEBUG }, 22:43:31 DEBUG "result": [ 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG [...] 22:43:31 DEBUG [...] 22:43:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:43:31 DEBUG [...] 22:43:31 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 22:43:31 DEBUG [...] 22:43:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:43:31 DEBUG [...] 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG [...] 22:43:31 DEBUG [...] 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG [...] 22:43:31 DEBUG 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:31 DEBUG "result": [ 22:43:31 DEBUG --- status code --- 22:43:31 DEBUG "value": [ 22:43:31 DEBUG { 22:43:31 DEBUG "result": [ 22:43:31 DEBUG { 22:43:31 DEBUG [...] 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG }, 22:43:31 DEBUG "value": [ 22:43:31 DEBUG }, 22:43:31 DEBUG 22:43:31 DEBUG }, 22:43:31 DEBUG 22:43:31 DEBUG "result": [ 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "result": [ 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG { 22:43:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:31 DEBUG 22:43:31 DEBUG "metric": { 22:43:31 DEBUG [...] 22:43:31 DEBUG "metric": { 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG { 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG [...] 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG [...] 22:43:31 DEBUG { 22:43:31 DEBUG { 22:43:31 DEBUG "metric": { 22:43:31 DEBUG --- http response --- 22:43:31 DEBUG 22:43:31 DEBUG "pod": "ds-cts-0" 22:43:31 DEBUG 22:43:31 DEBUG "pod": "ds-cts-0" 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 22:43:31 DEBUG }, 22:43:31 DEBUG 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "metric": { 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 22:43:31 DEBUG 22:43:31 DEBUG "instance": "10.142.0.43:9100" 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "metric": { 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG "pod": "ds-idrepo-0" 22:43:31 DEBUG 22:43:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:31 DEBUG }, 22:43:31 DEBUG 22:43:31 DEBUG }, 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG "0" 22:43:31 DEBUG "instance": "10.142.0.43:9100" 22:43:31 DEBUG 22:43:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:43:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:43:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:43:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:43:31 DEBUG }, 22:43:31 DEBUG 22:43:31 DEBUG }, 22:43:31 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 22:43:31 DEBUG 22:43:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:43:31 DEBUG "data": { 22:43:31 DEBUG }, 22:43:31 DEBUG [...] 22:43:31 DEBUG ----- output ----- 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG "value": [ 22:43:31 DEBUG "pod": "ds-idrepo-2" 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:43:31 DEBUG "0" 22:43:31 DEBUG ] 22:43:31 DEBUG }, 22:43:31 DEBUG 22:43:31 DEBUG }, 22:43:31 DEBUG }, 22:43:31 DEBUG }, 22:43:31 DEBUG }, 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG 22:43:31 DEBUG }, 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG { 22:43:31 DEBUG 22:43:31 DEBUG "pod": "ds-idrepo-2" 22:43:31 DEBUG 22:43:31 DEBUG }, 22:43:31 DEBUG "instance": "10.142.0.43:9100" 22:43:31 DEBUG [...] 22:43:31 DEBUG 22:43:31 DEBUG }, 22:43:31 DEBUG ] 22:43:31 DEBUG "value": [ 22:43:31 DEBUG [...] 22:43:31 DEBUG "value": [ 22:43:31 DEBUG "value": [ 22:43:31 DEBUG "value": [ 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG "pod": "ds-idrepo-2" 22:43:31 DEBUG }, 22:43:31 DEBUG "27717.966666666667" 22:43:31 DEBUG "pod": "ds-cts-2" 22:43:31 DEBUG "value": [ 22:43:31 DEBUG "result": [ 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "status": "success", 22:43:31 DEBUG 22:43:31 DEBUG }, 22:43:31 DEBUG 22:43:31 DEBUG "value": [ 22:43:31 DEBUG }, 22:43:31 DEBUG } 22:43:31 DEBUG 22:43:31 DEBUG "value": [ 22:43:31 DEBUG } 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG "0.00021102048229973977" 22:43:31 DEBUG }, 22:43:31 DEBUG "value": [ 22:43:31 DEBUG ] 22:43:31 DEBUG } 22:43:31 DEBUG 22:43:31 DEBUG { 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "data": { 22:43:31 DEBUG 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG "value": [ 22:43:31 DEBUG ] 22:43:31 DEBUG 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG ] 22:43:31 DEBUG "0.0003333333333333262" 22:43:31 DEBUG 22:43:31 DEBUG "1625423872" 22:43:31 DEBUG 22:43:31 DEBUG "0" 22:43:31 DEBUG "0" 22:43:31 DEBUG ] 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG ] 22:43:31 DEBUG }, 22:43:31 DEBUG 22:43:31 DEBUG "metric": { 22:43:31 DEBUG 22:43:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:43:31 DEBUG "resultType": "vector", 22:43:31 DEBUG [...] 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG [...] 22:43:31 DEBUG "0" 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG } 22:43:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:43:31 DEBUG }, 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG 22:43:31 DEBUG ] 22:43:31 DEBUG 22:43:31 DEBUG ] 22:43:31 DEBUG ] 22:43:31 DEBUG } 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG "0" 22:43:31 DEBUG ] 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG "instance": "10.142.0.107:9100" 22:43:31 DEBUG [...] 22:43:31 DEBUG }, 22:43:31 DEBUG "result": [ 22:43:31 DEBUG 22:43:31 DEBUG "0" 22:43:31 DEBUG 22:43:31 DEBUG ] 22:43:31 DEBUG "24509.13333333333" 22:43:31 DEBUG } 22:43:31 DEBUG "0.024279921706832183" 22:43:31 DEBUG "value": [ 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:43:31 DEBUG } 22:43:31 DEBUG [...] 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG "0" 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG [...] 22:43:31 DEBUG }, 22:43:31 DEBUG 22:43:31 DEBUG "value": [ 22:43:31 DEBUG { 22:43:31 DEBUG 22:43:31 DEBUG ] 22:43:31 DEBUG 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG ] 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG ] 22:43:31 DEBUG }, 22:43:31 DEBUG ] 22:43:31 DEBUG 22:43:31 DEBUG ] 22:43:31 DEBUG ] 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG "0" 22:43:31 DEBUG 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG "metric": { 22:43:31 DEBUG 22:43:31 DEBUG } 22:43:31 DEBUG 22:43:31 DEBUG ] 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG "0" 22:43:31 DEBUG } 22:43:31 DEBUG "value": [ 22:43:31 DEBUG } 22:43:31 DEBUG 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG "0" 22:43:31 DEBUG "instance": "10.142.0.107:9100" 22:43:31 DEBUG "pod": "ds-cts-2" 22:43:31 DEBUG ] 22:43:31 DEBUG "pod": "ds-cts-2" 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG ] 22:43:31 DEBUG ] 22:43:31 DEBUG } 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG } 22:43:31 DEBUG 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG 22:43:31 DEBUG ] 22:43:31 DEBUG }, 22:43:31 DEBUG }, 22:43:31 DEBUG } 22:43:31 DEBUG }, 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG "0" 22:43:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:43:31 DEBUG "pod": "ds-cts-2" 22:43:31 DEBUG [...] 22:43:31 DEBUG } 22:43:31 DEBUG "value": [ 22:43:31 DEBUG "value": [ 22:43:31 DEBUG } 22:43:31 DEBUG "value": [ 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG ] 22:43:31 DEBUG }, 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG }, 22:43:31 DEBUG }, 22:43:31 DEBUG 22:43:31 DEBUG ] 22:43:31 DEBUG 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG "value": [ 22:43:31 DEBUG } 22:43:31 DEBUG "value": [ 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG } 22:43:31 DEBUG 22:43:31 DEBUG "0" 22:43:31 DEBUG "0" 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG 22:43:31 DEBUG } 22:43:31 DEBUG 22:43:31 DEBUG ] 22:43:31 DEBUG ] 22:43:31 DEBUG } 22:43:31 DEBUG "0" 22:43:31 DEBUG "0.015" 22:43:31 DEBUG "0" 22:43:31 DEBUG "instance": "10.142.0.43:9100" 22:43:31 DEBUG [...] 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG ] 22:43:31 DEBUG ] 22:43:31 DEBUG }, 22:43:31 DEBUG 22:43:31 DEBUG ] 22:43:31 DEBUG ] 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG "value": [ 22:43:31 DEBUG 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG "157943768064" 22:43:31 DEBUG "instance": "10.142.0.43:9100" 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG }, 22:43:31 DEBUG } 22:43:31 DEBUG } 22:43:31 DEBUG "value": [ 22:43:31 DEBUG ] 22:43:31 DEBUG 1713390211, 22:43:31 DEBUG } 22:43:31 DEBUG "0.0005666666666684251" 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG } 22:43:31 DEBUG ] 22:43:31 DEBUG } 22:43:31 DEBUG } 22:44:31 INFO 22:44:31 INFO [loop_until]: kubectl --namespace=xlou top pods 22:44:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:44:31 INFO [loop_until]: OK (rc = 0) 22:44:31 DEBUG --- stdout --- 22:44:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 2904m 8446Mi am-b59c765b7-bgsxp 3022m 8634Mi am-b59c765b7-ch5q4 4058m 8285Mi ds-cts-0 1135m 7802Mi ds-cts-1 1562m 7586Mi ds-cts-2 2499m 7871Mi ds-idrepo-0 1856m 3143Mi ds-idrepo-1 1844m 2838Mi ds-idrepo-2 1492m 2766Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1010Mi lodemon-7cfb6bd874-prp2f 2m 61Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 3102m 1891Mi 22:44:31 DEBUG --- stderr --- 22:44:31 DEBUG 22:44:31 INFO 22:44:31 INFO [loop_until]: kubectl --namespace=xlou top node 22:44:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390271" 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 ERROR [http_cmd]: ERROR 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390271" 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 ERROR --- status code --- 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 DEBUG --- status code --- 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [loop_until]: OK (rc = 0) 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 DEBUG --- status code --- 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG --- status code --- 22:44:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- stdout --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 ERROR --- http response --- 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5441Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2119m 7% 6062Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 4234m 14% 10876Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3009Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1592m 5% 10016Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5001Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 4424m 14% 11077Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 11212m 37% 2924Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 3156m 10% 10476Mi 9% gke-xlou-cdm-ds-0f601751-j488 2454m 8% 5346Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 2406m 8% 11126Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 2182m 7% 5949Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 2609m 8% 9568Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 3584m 22% 5713Mi 9% 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG --- http response --- 22:44:31 INFO [http_cmd]: http status code OK 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG --- stderr --- 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG --- status code --- 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:31 DEBUG --- http response --- 22:44:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:31 DEBUG 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:44:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG { 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:31 DEBUG { 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG --- http response --- 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG { 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "data": { 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG { 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG { 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG { 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG { 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG "data": { 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG { 22:44:31 DEBUG "data": { 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "data": { 22:44:31 DEBUG "data": { 22:44:31 DEBUG ----- output ----- 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "data": { 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "data": { 22:44:31 DEBUG "data": { 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "data": { 22:44:31 DEBUG "data": { 22:44:31 DEBUG { 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "data": { 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "data": { 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "data": { 22:44:31 DEBUG "data": { 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG { 22:44:31 DEBUG "data": { 22:44:31 DEBUG "data": { 22:44:31 DEBUG "data": { 22:44:31 DEBUG "data": { 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "data": { 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "data": { 22:44:31 DEBUG { 22:44:31 DEBUG "data": { 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "data": { 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "data": { 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "status": "success", 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG { 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "data": { 22:44:31 DEBUG { 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG { 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG { 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "result": [ 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG "data": { 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "result": [ 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "instance": "10.142.0.107:9100" 22:44:31 DEBUG "result": [ 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG "metric": { 22:44:31 DEBUG { 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "metric": { 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "resultType": "vector", 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG { 22:44:31 DEBUG "instance": "10.142.0.107:9100" 22:44:31 DEBUG { 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "metric": { 22:44:31 DEBUG { 22:44:31 DEBUG }, 22:44:31 DEBUG { 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "result": [ 22:44:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:44:31 DEBUG "metric": { 22:44:31 DEBUG { 22:44:31 DEBUG "pod": "ds-idrepo-0" 22:44:31 DEBUG { 22:44:31 DEBUG "pod": "ds-cts-0" 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "pod": "ds-cts-0" 22:44:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:44:31 DEBUG "result": [ 22:44:31 DEBUG { 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "metric": { 22:44:31 DEBUG }, 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "instance": "10.142.0.107:9100" 22:44:31 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:44:31 DEBUG "instance": "10.142.0.107:9100" 22:44:31 DEBUG { 22:44:31 DEBUG }, 22:44:31 DEBUG "pod": "ds-idrepo-0" 22:44:31 DEBUG "metric": { 22:44:31 DEBUG }, 22:44:31 DEBUG "metric": { 22:44:31 DEBUG }, 22:44:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:44:31 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "pod": "ds-idrepo-0" 22:44:31 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:44:31 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "instance": "10.142.0.107:9100" 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:44:31 DEBUG 22:44:31 DEBUG "pod": "ds-idrepo-0" 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG "metric": { 22:44:31 DEBUG "value": [ 22:44:31 DEBUG }, 22:44:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:44:31 DEBUG "value": [ 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:44:31 DEBUG "pod": "ds-cts-0" 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG 22:44:31 DEBUG }, 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG }, 22:44:31 DEBUG 22:44:31 DEBUG }, 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:44:31 DEBUG }, 22:44:31 DEBUG "value": [ 22:44:31 DEBUG }, 22:44:31 DEBUG 22:44:31 DEBUG }, 22:44:31 DEBUG 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG [...] 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG [...] 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG [...] 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG [...] 22:44:31 DEBUG [...] 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG "instance": "10.142.0.43:9100" 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG "instance": "10.142.0.43:9100" 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG }, 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG "pod": "ds-cts-2" 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG "pod": "ds-cts-2" 22:44:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG }, 22:44:31 DEBUG 22:44:31 DEBUG "instance": "10.142.0.43:9100" 22:44:31 DEBUG "pod": "am-b59c765b7-7xg4f" 22:44:31 DEBUG 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 22:44:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:44:31 DEBUG "instance": "10.142.0.43:9100" 22:44:31 DEBUG 22:44:31 DEBUG }, 22:44:31 DEBUG "pod": "ds-idrepo-2" 22:44:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:44:31 DEBUG "pod": "ds-idrepo-2" 22:44:31 DEBUG 22:44:31 DEBUG }, 22:44:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:44:31 DEBUG "pod": "am-b59c765b7-7xg4f" 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG 22:44:31 DEBUG 22:44:31 DEBUG "pod": "ds-cts-2" 22:44:31 DEBUG "pod": "am-b59c765b7-7xg4f" 22:44:31 DEBUG "pod": "am-b59c765b7-7xg4f" 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "instance": "10.142.0.43:9100" 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG "pod": "ds-idrepo-2" 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG 22:44:31 DEBUG "value": [ 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:44:31 DEBUG "pod": "ds-cts-2" 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG }, 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG }, 22:44:31 DEBUG "value": [ 22:44:31 DEBUG }, 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG "1.1957524492474811" 22:44:31 DEBUG }, 22:44:31 DEBUG }, 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "157999686656" 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG "0.006666666666666672" 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG "452.59999999999997" 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG }, 22:44:31 DEBUG "2029432832" 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG }, 22:44:31 DEBUG "0" 22:44:31 DEBUG "0" 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG ] 22:44:31 DEBUG "value": [ 22:44:31 DEBUG "value": [ 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG ] 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG "6354031.199999999" 22:44:31 DEBUG "551.5941150725516" 22:44:31 DEBUG ] 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG ] 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG "6350660.066666666" 22:44:31 DEBUG "value": [ 22:44:31 DEBUG ] 22:44:31 DEBUG "0" 22:44:31 DEBUG ] 22:44:31 DEBUG "1.9804460478350976" 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG "453.06666666666666" 22:44:31 DEBUG "0.00004198666666666666" 22:44:31 DEBUG "6786" 22:44:31 DEBUG } 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG "0" 22:44:31 DEBUG "0" 22:44:31 DEBUG "489815.5741844259" 22:44:31 DEBUG } 22:44:31 DEBUG "0.009166666666666856" 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG "453.1" 22:44:31 DEBUG } 22:44:31 DEBUG "0" 22:44:31 DEBUG ] 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG } 22:44:31 DEBUG "value": [ 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG "0.015" 22:44:31 DEBUG "0" 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG "0" 22:44:31 DEBUG ] 22:44:31 DEBUG 1713390271, 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG "0" 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG ] 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:44:31 DEBUG } 22:45:31 INFO 22:45:31 INFO [loop_until]: kubectl --namespace=xlou top pods 22:45:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:45:31 INFO [loop_until]: OK (rc = 0) 22:45:31 DEBUG --- stdout --- 22:45:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 9411m 8454Mi am-b59c765b7-bgsxp 8641m 7533Mi am-b59c765b7-ch5q4 9418m 8290Mi ds-cts-0 5155m 7843Mi ds-cts-1 4882m 7644Mi ds-cts-2 5161m 7899Mi ds-idrepo-0 4631m 3122Mi ds-idrepo-1 4535m 2838Mi ds-idrepo-2 4586m 2767Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 965Mi idm-b85f9d978-wnhxb 4m 1010Mi lodemon-7cfb6bd874-prp2f 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6847m 1993Mi 22:45:31 DEBUG --- stderr --- 22:45:31 DEBUG 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:45:31 INFO 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390331" 22:45:31 INFO [loop_until]: kubectl --namespace=xlou top node 22:45:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:45:31 ERROR [http_cmd]: ERROR 22:45:31 ERROR --- status code --- 22:45:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:45:31 ERROR --- http response --- 22:45:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:45:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390331" 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 DEBUG --- status code --- 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [loop_until]: OK (rc = 0) 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- status code --- 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 DEBUG --- status code --- 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- stdout --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- status code --- 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 DEBUG --- status code --- 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 INFO [http_cmd]: http status code OK 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5440Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4525m 15% 6042Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9403m 31% 10862Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3000Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 5164m 17% 10076Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 4987Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9366m 31% 11074Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 28013m 93% 2910Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9265m 31% 9380Mi 8% gke-xlou-cdm-ds-0f601751-j488 4807m 16% 5336Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 4896m 16% 11173Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5117m 17% 5940Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 5188m 17% 9607Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6873m 43% 5805Mi 9% 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG --- status code --- 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- stderr --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:31 DEBUG 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG { 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:31 DEBUG { 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG --- http response --- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:31 DEBUG { 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:45:31 DEBUG { 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG { 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:45:31 DEBUG { 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "data": { 22:45:31 DEBUG { 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG "data": { 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "result": [ 22:45:31 DEBUG { 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG { 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG { 22:45:31 DEBUG "data": { 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "data": { 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "data": { 22:45:31 DEBUG "data": { 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG { 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG ----- output ----- 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "instance": "10.142.0.107:9100" 22:45:31 DEBUG "data": { 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "data": { 22:45:31 DEBUG "data": { 22:45:31 DEBUG "data": { 22:45:31 DEBUG "data": { 22:45:31 DEBUG "data": { 22:45:31 DEBUG "data": { 22:45:31 DEBUG "data": { 22:45:31 DEBUG "data": { 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG { 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "data": { 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "data": { 22:45:31 DEBUG "data": { 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "data": { 22:45:31 DEBUG { 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG }, 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "data": { 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "data": { 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG { 22:45:31 DEBUG "result": [ 22:45:31 DEBUG { 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "data": { 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG { 22:45:31 DEBUG "data": { 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "result": [ 22:45:31 DEBUG { 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "data": { 22:45:31 DEBUG "metric": { 22:45:31 DEBUG { 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "result": [ 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "status": "success", 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "value": [ 22:45:31 DEBUG { 22:45:31 DEBUG "result": [ 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "result": [ 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "pod": "ds-cts-0" 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "pod": "ds-cts-0" 22:45:31 DEBUG { 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "metric": { 22:45:31 DEBUG { 22:45:31 DEBUG { 22:45:31 DEBUG "result": [ 22:45:31 DEBUG { 22:45:31 DEBUG "data": { 22:45:31 DEBUG { 22:45:31 DEBUG 22:45:31 DEBUG "metric": { 22:45:31 DEBUG { 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "pod": "ds-idrepo-0" 22:45:31 DEBUG { 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "metric": { 22:45:31 DEBUG { 22:45:31 DEBUG "result": [ 22:45:31 DEBUG }, 22:45:31 DEBUG "instance": "10.142.0.107:9100" 22:45:31 DEBUG }, 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:45:31 DEBUG "instance": "10.142.0.107:9100" 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "metric": { 22:45:31 DEBUG { 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "resultType": "vector", 22:45:31 DEBUG "metric": { 22:45:31 DEBUG 22:45:31 DEBUG "pod": "ds-idrepo-0" 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:45:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:45:31 DEBUG "instance": "10.142.0.107:9100" 22:45:31 DEBUG }, 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:45:31 DEBUG "pod": "ds-idrepo-0" 22:45:31 DEBUG "pod": "ds-idrepo-0" 22:45:31 DEBUG "metric": { 22:45:31 DEBUG { 22:45:31 DEBUG "value": [ 22:45:31 DEBUG }, 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "pod": "ds-idrepo-0" 22:45:31 DEBUG }, 22:45:31 DEBUG }, 22:45:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:45:31 DEBUG "pod": "ds-idrepo-0" 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "instance": "10.142.0.107:9100" 22:45:31 DEBUG "result": [ 22:45:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:45:31 DEBUG 22:45:31 DEBUG }, 22:45:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:45:31 DEBUG }, 22:45:31 DEBUG }, 22:45:31 DEBUG }, 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:45:31 DEBUG }, 22:45:31 DEBUG }, 22:45:31 DEBUG }, 22:45:31 DEBUG "pod": "am-b59c765b7-ch5q4" 22:45:31 DEBUG "metric": { 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG }, 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "value": [ 22:45:31 DEBUG }, 22:45:31 DEBUG }, 22:45:31 DEBUG }, 22:45:31 DEBUG "pod": "ds-idrepo-0" 22:45:31 DEBUG { 22:45:31 DEBUG }, 22:45:31 DEBUG [...] 22:45:31 DEBUG "value": [ 22:45:31 DEBUG }, 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "value": [ 22:45:31 DEBUG 22:45:31 DEBUG }, 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "value": [ 22:45:31 DEBUG }, 22:45:31 DEBUG "pod": "ds-idrepo-0" 22:45:31 DEBUG "value": [ 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "value": [ 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "value": [ 22:45:31 DEBUG }, 22:45:31 DEBUG "metric": { 22:45:31 DEBUG "value": [ 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "value": [ 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "value": [ 22:45:31 DEBUG }, 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "pod": "ds-cts-0" 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "value": [ 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "value": [ 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG }, 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "value": [ 22:45:31 DEBUG 22:45:31 DEBUG "instance": "10.142.0.43:9100" 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG [...] 22:45:31 DEBUG [...] 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG }, 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "pod": "ds-cts-2" 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:45:31 DEBUG 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG }, 22:45:31 DEBUG "pod": "ds-cts-2" 22:45:31 DEBUG 22:45:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "value": [ 22:45:31 DEBUG [...] 22:45:31 DEBUG 22:45:31 DEBUG "pod": "ds-idrepo-2" 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "instance": "10.142.0.43:9100" 22:45:31 DEBUG "pod": "ds-idrepo-2" 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG }, 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "value": [ 22:45:31 DEBUG }, 22:45:31 DEBUG 22:45:31 DEBUG }, 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "pod": "ds-idrepo-2" 22:45:31 DEBUG 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG }, 22:45:31 DEBUG 22:45:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:45:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:45:31 DEBUG 22:45:31 DEBUG }, 22:45:31 DEBUG }, 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "value": [ 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG 22:45:31 DEBUG "instance": "10.142.0.43:9100" 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "pod": "ds-idrepo-2" 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "instance": "10.142.0.43:9100" 22:45:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:45:31 DEBUG 22:45:31 DEBUG }, 22:45:31 DEBUG 22:45:31 DEBUG "48375006.7" 22:45:31 DEBUG 22:45:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:45:31 DEBUG "value": [ 22:45:31 DEBUG 22:45:31 DEBUG }, 22:45:31 DEBUG }, 22:45:31 DEBUG 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "pod": "ds-cts-2" 22:45:31 DEBUG "pod": "ds-idrepo-2" 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG "pod": "ds-idrepo-2" 22:45:31 DEBUG }, 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG }, 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG }, 22:45:31 DEBUG }, 22:45:31 DEBUG "instance": "10.142.0.43:9100" 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "pod": "ds-idrepo-2" 22:45:31 DEBUG ] 22:45:31 DEBUG 22:45:31 DEBUG }, 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG "0" 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG }, 22:45:31 DEBUG }, 22:45:31 DEBUG "1840.7333333333333" 22:45:31 DEBUG }, 22:45:31 DEBUG "0.00002099333333333333" 22:45:31 DEBUG }, 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "0" 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "1840.7666666666667" 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "value": [ 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG "0" 22:45:31 DEBUG } 22:45:31 DEBUG "pod": "ds-cts-2" 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "pod": "am-b59c765b7-bgsxp" 22:45:31 DEBUG ] 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG }, 22:45:31 DEBUG "48377500.46666667" 22:45:31 DEBUG "2432.9405191528917" 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "value": [ 22:45:31 DEBUG ] 22:45:31 DEBUG "value": [ 22:45:31 DEBUG ] 22:45:31 DEBUG "value": [ 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG ] 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG ] 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG }, 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "6.626269770997017" 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG }, 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG }, 22:45:31 DEBUG } 22:45:31 DEBUG "2149482496" 22:45:31 DEBUG "93609.5" 22:45:31 DEBUG "value": [ 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG } 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG } 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG "159481273344" 22:45:31 DEBUG } 22:45:31 DEBUG "2.9849433401646484" 22:45:31 DEBUG } 22:45:31 DEBUG "0.028333333333333443" 22:45:31 DEBUG }, 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "0.016" 22:45:31 DEBUG "value": [ 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG "0" 22:45:31 DEBUG "0" 22:45:31 DEBUG ] 22:45:31 DEBUG "1840.6333333333334" 22:45:31 DEBUG ] 22:45:31 DEBUG "94813807.63183962" 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG "value": [ 22:45:31 DEBUG "0.03933333333333356" 22:45:31 DEBUG } 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG ] 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG "0" 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG 1713390331, 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG "0" 22:45:31 DEBUG } 22:45:31 DEBUG "0" 22:45:31 DEBUG } 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG ] 22:45:31 DEBUG "0" 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG ] 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:45:31 DEBUG } 22:46:31 INFO 22:46:31 INFO [loop_until]: kubectl --namespace=xlou top pods 22:46:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [loop_until]: OK (rc = 0) 22:46:31 DEBUG --- stdout --- 22:46:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 9443m 8456Mi am-b59c765b7-bgsxp 8318m 7534Mi am-b59c765b7-ch5q4 9558m 8289Mi ds-cts-0 5095m 7848Mi ds-cts-1 4939m 7656Mi ds-cts-2 5197m 7924Mi ds-idrepo-0 4788m 3172Mi ds-idrepo-1 4617m 2838Mi ds-idrepo-2 4694m 2855Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 3m 1010Mi lodemon-7cfb6bd874-prp2f 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6590m 1997Mi 22:46:31 DEBUG --- stderr --- 22:46:31 DEBUG 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390391" 22:46:31 INFO [http_cmd]: http status code OK 22:46:31 DEBUG --- status code --- 22:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:31 DEBUG --- http response --- 22:46:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:46:31 ERROR [http_cmd]: ERROR 22:46:31 ERROR --- status code --- 22:46:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:46:31 ERROR --- http response --- 22:46:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:46:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:46:31 INFO 22:46:31 INFO [loop_until]: kubectl --namespace=xlou top node 22:46:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390391" 22:46:31 INFO [http_cmd]: http status code OK 22:46:31 INFO [http_cmd]: http status code OK 22:46:31 INFO [http_cmd]: http status code OK 22:46:31 INFO [http_cmd]: http status code OK 22:46:31 INFO [http_cmd]: http status code OK 22:46:31 INFO [http_cmd]: http status code OK 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 INFO [loop_until]: OK (rc = 0) 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 DEBUG --- status code --- 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG --- status code --- 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 DEBUG --- status code --- 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG --- stdout --- 22:46:32 DEBUG --- status code --- 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5434Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4662m 15% 6099Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9141m 30% 10871Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 2993Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 5180m 17% 10117Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 4989Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9377m 31% 11055Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 27880m 93% 2940Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9212m 30% 9373Mi 8% gke-xlou-cdm-ds-0f601751-j488 4931m 16% 5414Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 4943m 16% 11231Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5055m 16% 5944Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 5133m 17% 9642Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6743m 42% 5802Mi 9% 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 INFO [http_cmd]: http status code OK 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG --- stderr --- 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG --- status code --- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:32 DEBUG 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:46:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG { 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG --- http response --- 22:46:32 DEBUG { 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG { 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG { 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG { 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "data": { 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG { 22:46:32 DEBUG "data": { 22:46:32 DEBUG { 22:46:32 DEBUG ----- output ----- 22:46:32 DEBUG "data": { 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "data": { 22:46:32 DEBUG "data": { 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "data": { 22:46:32 DEBUG "data": { 22:46:32 DEBUG "data": { 22:46:32 DEBUG "data": { 22:46:32 DEBUG "data": { 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "data": { 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "data": { 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "data": { 22:46:32 DEBUG "data": { 22:46:32 DEBUG "data": { 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG { 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "data": { 22:46:32 DEBUG "data": { 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "data": { 22:46:32 DEBUG "data": { 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "data": { 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "data": { 22:46:32 DEBUG "data": { 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "data": { 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "data": { 22:46:32 DEBUG "status": "success", 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG { 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "result": [ 22:46:32 DEBUG { 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "result": [ 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG { 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "data": { 22:46:32 DEBUG { 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "result": [ 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG "result": [ 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG { 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "resultType": "vector", 22:46:32 DEBUG "metric": { 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG "instance": "10.142.0.107:9100" 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:46:32 DEBUG { 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "metric": { 22:46:32 DEBUG { 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:46:32 DEBUG { 22:46:32 DEBUG }, 22:46:32 DEBUG { 22:46:32 DEBUG { 22:46:32 DEBUG "pod": "ds-idrepo-0" 22:46:32 DEBUG }, 22:46:32 DEBUG { 22:46:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "metric": { 22:46:32 DEBUG }, 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "metric": { 22:46:32 DEBUG }, 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "pod": "ds-cts-0" 22:46:32 DEBUG "pod": "ds-idrepo-1" 22:46:32 DEBUG "instance": "10.142.0.107:9100" 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "pod": "ds-idrepo-0" 22:46:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:46:32 DEBUG { 22:46:32 DEBUG "pod": "ds-idrepo-1" 22:46:32 DEBUG }, 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "metric": { 22:46:32 DEBUG "result": [ 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "metric": { 22:46:32 DEBUG }, 22:46:32 DEBUG "instance": "10.142.0.107:9100" 22:46:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "instance": "10.142.0.107:9100" 22:46:32 DEBUG "pod": "ds-idrepo-0" 22:46:32 DEBUG "pod": "ds-idrepo-0" 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "pod": "ds-cts-0" 22:46:32 DEBUG }, 22:46:32 DEBUG }, 22:46:32 DEBUG }, 22:46:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:46:32 DEBUG }, 22:46:32 DEBUG }, 22:46:32 DEBUG "metric": { 22:46:32 DEBUG }, 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:46:32 DEBUG }, 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG { 22:46:32 DEBUG 22:46:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:46:32 DEBUG "value": [ 22:46:32 DEBUG }, 22:46:32 DEBUG }, 22:46:32 DEBUG 22:46:32 DEBUG }, 22:46:32 DEBUG }, 22:46:32 DEBUG }, 22:46:32 DEBUG 22:46:32 DEBUG }, 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "value": [ 22:46:32 DEBUG }, 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "pod": "ds-idrepo-1" 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG "instance": "10.142.0.107:9100" 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "pod": "ds-cts-0" 22:46:32 DEBUG 22:46:32 DEBUG "metric": { 22:46:32 DEBUG 22:46:32 DEBUG }, 22:46:32 DEBUG 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG }, 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG }, 22:46:32 DEBUG 22:46:32 DEBUG }, 22:46:32 DEBUG [...] 22:46:32 DEBUG "pod": "ds-idrepo-1" 22:46:32 DEBUG 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG }, 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:46:32 DEBUG [...] 22:46:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG }, 22:46:32 DEBUG 22:46:32 DEBUG }, 22:46:32 DEBUG 22:46:32 DEBUG "pod": "ds-idrepo-2" 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG "instance": "10.142.0.43:9100" 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:46:32 DEBUG 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG "value": [ 22:46:32 DEBUG [...] 22:46:32 DEBUG }, 22:46:32 DEBUG 22:46:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:46:32 DEBUG 22:46:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:46:32 DEBUG "instance": "10.142.0.43:9100" 22:46:32 DEBUG 22:46:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:46:32 DEBUG "pod": "ds-cts-2" 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG "pod": "ds-cts-2" 22:46:32 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:46:32 DEBUG }, 22:46:32 DEBUG 22:46:32 DEBUG "pod": "ds-idrepo-2" 22:46:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:46:32 DEBUG 22:46:32 DEBUG 22:46:32 DEBUG }, 22:46:32 DEBUG 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG }, 22:46:32 DEBUG "value": [ 22:46:32 DEBUG }, 22:46:32 DEBUG }, 22:46:32 DEBUG "value": [ 22:46:32 DEBUG }, 22:46:32 DEBUG }, 22:46:32 DEBUG "pod": "ds-idrepo-2" 22:46:32 DEBUG "pod": "ds-cts-2" 22:46:32 DEBUG }, 22:46:32 DEBUG }, 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG }, 22:46:32 DEBUG }, 22:46:32 DEBUG 22:46:32 DEBUG [...] 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:46:32 DEBUG "203348" 22:46:32 DEBUG "instance": "10.142.0.43:9100" 22:46:32 DEBUG "6.62843791353117" 22:46:32 DEBUG 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:46:32 DEBUG "instance": "10.142.0.43:9100" 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "instance": "10.142.0.43:9100" 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "value": [ 22:46:32 DEBUG }, 22:46:32 DEBUG }, 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:46:32 DEBUG 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG }, 22:46:32 DEBUG ] 22:46:32 DEBUG }, 22:46:32 DEBUG ] 22:46:32 DEBUG 22:46:32 DEBUG "0" 22:46:32 DEBUG }, 22:46:32 DEBUG }, 22:46:32 DEBUG "1811.4666666666667" 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG }, 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG "48178086.83333333" 22:46:32 DEBUG }, 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG }, 22:46:32 DEBUG 22:46:32 DEBUG "0" 22:46:32 DEBUG "value": [ 22:46:32 DEBUG } 22:46:32 DEBUG "value": [ 22:46:32 DEBUG } 22:46:32 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:46:32 DEBUG ] 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "value": [ 22:46:32 DEBUG ] 22:46:32 DEBUG "0" 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG "48175763.96666667" 22:46:32 DEBUG "1811.4666666666667" 22:46:32 DEBUG "0" 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG "0" 22:46:32 DEBUG "0" 22:46:32 DEBUG ] 22:46:32 DEBUG "value": [ 22:46:32 DEBUG "0" 22:46:32 DEBUG "1811.4333333333334" 22:46:32 DEBUG "value": [ 22:46:32 DEBUG 22:46:32 DEBUG ] 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG ] 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG ] 22:46:32 DEBUG }, 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG "0.03893333333333355" 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG "0" 22:46:32 DEBUG "0.00012163333333333331" 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG "pod": "ds-cts-2" 22:46:32 DEBUG } 22:46:32 DEBUG "0.016" 22:46:32 DEBUG } 22:46:32 DEBUG "0.02833333333333332" 22:46:32 DEBUG } 22:46:32 DEBUG "value": [ 22:46:32 DEBUG } 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG "2192769024" 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG "0" 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG }, 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG } 22:46:32 DEBUG "161599093760" 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG "0" 22:46:32 DEBUG "value": [ 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG "0" 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG 1713390391, 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG "0" 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG ] 22:46:32 DEBUG } 22:46:32 DEBUG } 22:46:32 DEBUG } 22:47:31 INFO 22:47:31 INFO [loop_until]: kubectl --namespace=xlou top pods 22:47:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390451" 22:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390451" 22:47:31 INFO [http_cmd]: http status code OK 22:47:31 DEBUG --- status code --- 22:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:31 DEBUG --- http response --- 22:47:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:47:31 ERROR [http_cmd]: ERROR 22:47:31 ERROR --- status code --- 22:47:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:47:31 ERROR --- http response --- 22:47:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:47:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:47:31 INFO [loop_until]: OK (rc = 0) 22:47:31 DEBUG --- stdout --- 22:47:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 9923m 8461Mi am-b59c765b7-bgsxp 8952m 7543Mi am-b59c765b7-ch5q4 10264m 8861Mi ds-cts-0 8864m 7854Mi ds-cts-1 7049m 7661Mi ds-cts-2 8321m 7930Mi ds-idrepo-0 4999m 3183Mi ds-idrepo-1 4707m 2838Mi ds-idrepo-2 4913m 2905Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1010Mi lodemon-7cfb6bd874-prp2f 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6525m 2002Mi 22:47:31 DEBUG --- stderr --- 22:47:31 DEBUG 22:47:32 INFO 22:47:32 INFO [loop_until]: kubectl --namespace=xlou top node 22:47:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390452" 22:47:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390452" 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 DEBUG --- status code --- 22:47:32 INFO [loop_until]: OK (rc = 0) 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 DEBUG --- status code --- 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 DEBUG --- status code --- 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 DEBUG --- status code --- 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 DEBUG --- status code --- 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG --- status code --- 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG --- stdout --- 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5442Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4799m 16% 6102Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10548m 35% 11438Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 95m 0% 3002Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 7586m 25% 10079Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 164m 0% 4987Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10639m 35% 11073Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 27959m 93% 2921Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10282m 34% 9379Mi 8% gke-xlou-cdm-ds-0f601751-j488 5283m 17% 5473Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 8115m 27% 11193Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5409m 18% 5931Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 9106m 30% 9602Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6749m 42% 5804Mi 9% 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 INFO [http_cmd]: http status code OK 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:32 DEBUG --- stderr --- 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG --- status code --- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:47:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG { 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG --- http response --- 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG { 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG { 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG { 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG { 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG { 22:47:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG { 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG { 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "data": { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG ----- output ----- 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "data": { 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "data": { 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "data": { 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "data": { 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "data": { 22:47:32 DEBUG "data": { 22:47:32 DEBUG "data": { 22:47:32 DEBUG "data": { 22:47:32 DEBUG "data": { 22:47:32 DEBUG "data": { 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "data": { 22:47:32 DEBUG { 22:47:32 DEBUG "data": { 22:47:32 DEBUG "data": { 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "data": { 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "data": { 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "data": { 22:47:32 DEBUG "data": { 22:47:32 DEBUG "data": { 22:47:32 DEBUG "data": { 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "data": { 22:47:32 DEBUG "data": { 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "data": { 22:47:32 DEBUG "data": { 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "status": "success", 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "result": [ 22:47:32 DEBUG { 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "data": { 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "result": [ 22:47:32 DEBUG { 22:47:32 DEBUG "result": [ 22:47:32 DEBUG { 22:47:32 DEBUG "result": [ 22:47:32 DEBUG { 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "result": [ 22:47:32 DEBUG { 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "result": [ 22:47:32 DEBUG { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "result": [ 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG { 22:47:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:47:32 DEBUG { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "resultType": "vector", 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "pod": "ds-idrepo-0" 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "pod": "ds-idrepo-1" 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "metric": { 22:47:32 DEBUG }, 22:47:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:47:32 DEBUG "pod": "ds-idrepo-0" 22:47:32 DEBUG "instance": "10.142.0.107:9100" 22:47:32 DEBUG "pod": "ds-cts-0" 22:47:32 DEBUG }, 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "instance": "10.142.0.107:9100" 22:47:32 DEBUG "result": [ 22:47:32 DEBUG "pod": "ds-idrepo-1" 22:47:32 DEBUG "instance": "10.142.0.107:9100" 22:47:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:47:32 DEBUG }, 22:47:32 DEBUG "pod": "ds-idrepo-0" 22:47:32 DEBUG }, 22:47:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:47:32 DEBUG }, 22:47:32 DEBUG "pod": "ds-idrepo-1" 22:47:32 DEBUG "pod": "ds-cts-0" 22:47:32 DEBUG "instance": "10.142.0.107:9100" 22:47:32 DEBUG "pod": "ds-cts-0" 22:47:32 DEBUG }, 22:47:32 DEBUG "pod": "ds-idrepo-0" 22:47:32 DEBUG "pod": "ds-idrepo-1" 22:47:32 DEBUG "instance": "10.142.0.107:9100" 22:47:32 DEBUG "value": [ 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:47:32 DEBUG }, 22:47:32 DEBUG { 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG "value": [ 22:47:32 DEBUG 22:47:32 DEBUG "value": [ 22:47:32 DEBUG }, 22:47:32 DEBUG "value": [ 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG "value": [ 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG 22:47:32 DEBUG }, 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "metric": { 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG }, 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG "value": [ 22:47:32 DEBUG 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG "value": [ 22:47:32 DEBUG 22:47:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG "value": [ 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG }, 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG "value": [ 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG [...] 22:47:32 DEBUG [...] 22:47:32 DEBUG [...] 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG [...] 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG [...] 22:47:32 DEBUG [...] 22:47:32 DEBUG [...] 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG "pod": "ds-idrepo-2" 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG [...] 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG }, 22:47:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:47:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG 22:47:32 DEBUG "instance": "10.142.0.43:9100" 22:47:32 DEBUG 22:47:32 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:47:32 DEBUG }, 22:47:32 DEBUG 22:47:32 DEBUG }, 22:47:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:47:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:47:32 DEBUG "pod": "ds-idrepo-2" 22:47:32 DEBUG "instance": "10.142.0.43:9100" 22:47:32 DEBUG "pod": "ds-cts-2" 22:47:32 DEBUG }, 22:47:32 DEBUG 22:47:32 DEBUG "instance": "10.142.0.43:9100" 22:47:32 DEBUG 22:47:32 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:47:32 DEBUG "instance": "10.142.0.43:9100" 22:47:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:47:32 DEBUG 22:47:32 DEBUG "value": [ 22:47:32 DEBUG }, 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:47:32 DEBUG "pod": "ds-cts-2" 22:47:32 DEBUG 22:47:32 DEBUG }, 22:47:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:47:32 DEBUG 22:47:32 DEBUG "value": [ 22:47:32 DEBUG 22:47:32 DEBUG "value": [ 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:47:32 DEBUG }, 22:47:32 DEBUG 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG "pod": "ds-cts-2" 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG }, 22:47:32 DEBUG "pod": "ds-cts-2" 22:47:32 DEBUG "value": [ 22:47:32 DEBUG }, 22:47:32 DEBUG "pod": "ds-idrepo-2" 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG "instance": "10.142.0.43:9100" 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG }, 22:47:32 DEBUG "value": [ 22:47:32 DEBUG 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG }, 22:47:32 DEBUG "0" 22:47:32 DEBUG "6.585102093204368" 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "value": [ 22:47:32 DEBUG }, 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG "value": [ 22:47:32 DEBUG }, 22:47:32 DEBUG "0.00003414454524152314" 22:47:32 DEBUG }, 22:47:32 DEBUG "0" 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG "0" 22:47:32 DEBUG "value": [ 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG "value": [ 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG "value": [ 22:47:32 DEBUG "0.04300000000000163" 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG "value": [ 22:47:32 DEBUG ] 22:47:32 DEBUG "value": [ 22:47:32 DEBUG ] 22:47:32 DEBUG "452.8" 22:47:32 DEBUG "305955" 22:47:32 DEBUG "0" 22:47:32 DEBUG "49725328" 22:47:32 DEBUG "0" 22:47:32 DEBUG ] 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG "0.029666666666666758" 22:47:32 DEBUG }, 22:47:32 DEBUG "0" 22:47:32 DEBUG "163768017920" 22:47:32 DEBUG "1823.9666666666667" 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG "1824.4" 22:47:32 DEBUG "0" 22:47:32 DEBUG "0.00010286222222222221" 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG ] 22:47:32 DEBUG "1824.4333333333334" 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG } 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG "0.016" 22:47:32 DEBUG ] 22:47:32 DEBUG "value": [ 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG "0" 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG "0" 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG "0" 22:47:32 DEBUG ] 22:47:32 DEBUG "49727690.733333334" 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG 1713390452, 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG "2194964480" 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG ] 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:47:32 DEBUG } 22:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390511" 22:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390511" 22:48:31 INFO [http_cmd]: http status code OK 22:48:31 DEBUG --- status code --- 22:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:31 DEBUG --- http response --- 22:48:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:48:31 ERROR [http_cmd]: ERROR 22:48:31 ERROR --- status code --- 22:48:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:48:31 ERROR --- http response --- 22:48:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:48:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:48:32 INFO 22:48:32 INFO [loop_until]: kubectl --namespace=xlou top pods 22:48:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:48:32 INFO [loop_until]: OK (rc = 0) 22:48:32 DEBUG --- stdout --- 22:48:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11728m 8473Mi am-b59c765b7-bgsxp 11031m 7549Mi am-b59c765b7-ch5q4 12055m 8533Mi ds-cts-0 12166m 7834Mi ds-cts-1 11180m 7638Mi ds-cts-2 12604m 7934Mi ds-idrepo-0 5300m 3193Mi ds-idrepo-1 5317m 2839Mi ds-idrepo-2 5351m 2929Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1010Mi lodemon-7cfb6bd874-prp2f 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6743m 2002Mi 22:48:32 DEBUG --- stderr --- 22:48:32 DEBUG 22:48:32 INFO 22:48:32 INFO [loop_until]: kubectl --namespace=xlou top node 22:48:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390512" 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 DEBUG --- status code --- 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390512" 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [loop_until]: OK (rc = 0) 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390512" 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 DEBUG --- status code --- 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 DEBUG --- status code --- 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- stdout --- 22:48:32 DEBUG --- status code --- 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 DEBUG --- status code --- 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 59m 0% 5432Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5244m 17% 6108Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12003m 40% 11105Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3009Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12232m 40% 10069Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 4975Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11486m 38% 11079Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 28249m 94% 2950Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 12008m 40% 9389Mi 8% gke-xlou-cdm-ds-0f601751-j488 5696m 19% 5511Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11242m 37% 11171Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5880m 19% 5943Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12454m 41% 9598Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6795m 42% 5805Mi 9% 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- stderr --- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 INFO [http_cmd]: http status code OK 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:48:32 DEBUG { 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG --- status code --- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:32 DEBUG { 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG { 22:48:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:32 DEBUG { 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG { 22:48:32 DEBUG { 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG { 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG { 22:48:32 DEBUG { 22:48:32 DEBUG { 22:48:32 DEBUG { 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:32 DEBUG { 22:48:32 DEBUG { 22:48:32 DEBUG { 22:48:32 DEBUG { 22:48:32 DEBUG { 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:32 DEBUG { 22:48:32 DEBUG "data": { 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG --- http response --- 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG { 22:48:32 DEBUG "data": { 22:48:32 DEBUG { 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG { 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG { 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "data": { 22:48:32 DEBUG { 22:48:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:32 DEBUG "data": { 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "data": { 22:48:32 DEBUG "data": { 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "data": { 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "data": { 22:48:32 DEBUG "data": { 22:48:32 DEBUG "data": { 22:48:32 DEBUG "data": { 22:48:32 DEBUG { 22:48:32 DEBUG "data": { 22:48:32 DEBUG "data": { 22:48:32 DEBUG "data": { 22:48:32 DEBUG "data": { 22:48:32 DEBUG "data": { 22:48:32 DEBUG { 22:48:32 DEBUG "data": { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG ----- output ----- 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "data": { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "data": { 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "data": { 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "data": { 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "data": { 22:48:32 DEBUG { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG { 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "data": { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "data": { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "metric": { 22:48:32 DEBUG { 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "status": "success", 22:48:32 DEBUG { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "metric": { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG { 22:48:32 DEBUG { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "metric": { 22:48:32 DEBUG { 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG { 22:48:32 DEBUG { 22:48:32 DEBUG { 22:48:32 DEBUG { 22:48:32 DEBUG { 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG { 22:48:32 DEBUG "pod": "ds-idrepo-0" 22:48:32 DEBUG "metric": { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "data": { 22:48:32 DEBUG "metric": { 22:48:32 DEBUG { 22:48:32 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:48:32 DEBUG { 22:48:32 DEBUG "metric": { 22:48:32 DEBUG "metric": { 22:48:32 DEBUG { 22:48:32 DEBUG "metric": { 22:48:32 DEBUG { 22:48:32 DEBUG "metric": { 22:48:32 DEBUG { 22:48:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:48:32 DEBUG "metric": { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "metric": { 22:48:32 DEBUG "metric": { 22:48:32 DEBUG "metric": { 22:48:32 DEBUG "metric": { 22:48:32 DEBUG "metric": { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG "metric": { 22:48:32 DEBUG }, 22:48:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:48:32 DEBUG { 22:48:32 DEBUG "resultType": "vector", 22:48:32 DEBUG "pod": "ds-cts-0" 22:48:32 DEBUG "metric": { 22:48:32 DEBUG }, 22:48:32 DEBUG "metric": { 22:48:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:48:32 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:48:32 DEBUG "metric": { 22:48:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:48:32 DEBUG "metric": { 22:48:32 DEBUG "instance": "10.142.0.107:9100" 22:48:32 DEBUG "metric": { 22:48:32 DEBUG }, 22:48:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:48:32 DEBUG { 22:48:32 DEBUG "pod": "ds-cts-0" 22:48:32 DEBUG "pod": "ds-idrepo-0" 22:48:32 DEBUG "instance": "10.142.0.107:9100" 22:48:32 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:48:32 DEBUG "instance": "10.142.0.107:9100" 22:48:32 DEBUG { 22:48:32 DEBUG "pod": "ds-cts-0" 22:48:32 DEBUG "value": [ 22:48:32 DEBUG }, 22:48:32 DEBUG "metric": { 22:48:32 DEBUG "result": [ 22:48:32 DEBUG }, 22:48:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:48:32 DEBUG }, 22:48:32 DEBUG }, 22:48:32 DEBUG "pod": "ds-idrepo-0" 22:48:32 DEBUG }, 22:48:32 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:48:32 DEBUG }, 22:48:32 DEBUG "instance": "10.142.0.107:9100" 22:48:32 DEBUG "value": [ 22:48:32 DEBUG }, 22:48:32 DEBUG "metric": { 22:48:32 DEBUG }, 22:48:32 DEBUG }, 22:48:32 DEBUG }, 22:48:32 DEBUG }, 22:48:32 DEBUG }, 22:48:32 DEBUG "metric": { 22:48:32 DEBUG }, 22:48:32 DEBUG 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "pod": "ds-idrepo-0" 22:48:32 DEBUG { 22:48:32 DEBUG "value": [ 22:48:32 DEBUG }, 22:48:32 DEBUG 22:48:32 DEBUG }, 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "value": [ 22:48:32 DEBUG }, 22:48:32 DEBUG "value": [ 22:48:32 DEBUG }, 22:48:32 DEBUG "value": [ 22:48:32 DEBUG }, 22:48:32 DEBUG 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "instance": "10.142.0.107:9100" 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:48:32 DEBUG "value": [ 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG }, 22:48:32 DEBUG "metric": { 22:48:32 DEBUG 22:48:32 DEBUG "value": [ 22:48:32 DEBUG 22:48:32 DEBUG "value": [ 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG "value": [ 22:48:32 DEBUG 22:48:32 DEBUG "value": [ 22:48:32 DEBUG 22:48:32 DEBUG "value": [ 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG }, 22:48:32 DEBUG "value": [ 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG }, 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG }, 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG "value": [ 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG "value": [ 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG [...] 22:48:32 DEBUG [...] 22:48:32 DEBUG [...] 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG "pod": "ds-idrepo-2" 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG "pod": "ds-idrepo-2" 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG }, 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:48:32 DEBUG 22:48:32 DEBUG "pod": "ds-cts-2" 22:48:32 DEBUG 22:48:32 DEBUG "instance": "10.142.0.43:9100" 22:48:32 DEBUG 22:48:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG }, 22:48:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:48:32 DEBUG 22:48:32 DEBUG [...] 22:48:32 DEBUG "pod": "ds-cts-2" 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG 22:48:32 DEBUG }, 22:48:32 DEBUG }, 22:48:32 DEBUG 22:48:32 DEBUG }, 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG }, 22:48:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:48:32 DEBUG 22:48:32 DEBUG "pod": "ds-cts-2" 22:48:32 DEBUG "pod": "ds-idrepo-2" 22:48:32 DEBUG "instance": "10.142.0.43:9100" 22:48:32 DEBUG "pod": "ds-idrepo-2" 22:48:32 DEBUG "instance": "10.142.0.43:9100" 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG "value": [ 22:48:32 DEBUG }, 22:48:32 DEBUG 22:48:32 DEBUG 22:48:32 DEBUG }, 22:48:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:48:32 DEBUG 22:48:32 DEBUG "3987.9514406547605" 22:48:32 DEBUG 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:48:32 DEBUG "value": [ 22:48:32 DEBUG 22:48:32 DEBUG "instance": "10.142.0.43:9100" 22:48:32 DEBUG "value": [ 22:48:32 DEBUG }, 22:48:32 DEBUG 22:48:32 DEBUG }, 22:48:32 DEBUG }, 22:48:32 DEBUG }, 22:48:32 DEBUG }, 22:48:32 DEBUG }, 22:48:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:48:32 DEBUG "pod": "ds-cts-2" 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG "0" 22:48:32 DEBUG "pod": "ds-idrepo-2" 22:48:32 DEBUG }, 22:48:32 DEBUG "value": [ 22:48:32 DEBUG }, 22:48:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:48:32 DEBUG ] 22:48:32 DEBUG "pod": "ds-idrepo-2" 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG }, 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG 22:48:32 DEBUG "pod": "ds-idrepo-2" 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG }, 22:48:32 DEBUG "instance": "10.142.0.43:9100" 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "value": [ 22:48:32 DEBUG }, 22:48:32 DEBUG }, 22:48:32 DEBUG ] 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "value": [ 22:48:32 DEBUG 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG "value": [ 22:48:32 DEBUG }, 22:48:32 DEBUG } 22:48:32 DEBUG }, 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "0" 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "166162213888" 22:48:32 DEBUG }, 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "1823.0666666666666" 22:48:32 DEBUG "value": [ 22:48:32 DEBUG }, 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "value": [ 22:48:32 DEBUG } 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG "0" 22:48:32 DEBUG "0" 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG "value": [ 22:48:32 DEBUG ] 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG "1823.6333333333332" 22:48:32 DEBUG ] 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG ] 22:48:32 DEBUG "value": [ 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG ] 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG "value": [ 22:48:32 DEBUG "0" 22:48:32 DEBUG "0" 22:48:32 DEBUG "61015134.9" 22:48:32 DEBUG "302461.1920771595" 22:48:32 DEBUG "0.07833333333333312" 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG ] 22:48:32 DEBUG "0" 22:48:32 DEBUG 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG "0.016" 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG } 22:48:32 DEBUG "0" 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG "1823.2666666666667" 22:48:32 DEBUG } 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG "328281" 22:48:32 DEBUG } 22:48:32 DEBUG "0.1619333333333344" 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG "2197889024" 22:48:32 DEBUG "0.000043097777777777775" 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG "6.587445410929875" 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG "5.278264125110257" 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG "61017333.5" 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG }, 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG "value": [ 22:48:32 DEBUG 1713390512, 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG "1840.7" 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG } 22:48:32 DEBUG ] 22:48:32 DEBUG } 22:48:32 DEBUG } 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390572" 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:49:32 ERROR [http_cmd]: ERROR 22:49:32 ERROR --- status code --- 22:49:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:49:32 ERROR --- http response --- 22:49:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:49:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:49:32 INFO 22:49:32 INFO [loop_until]: kubectl --namespace=xlou top pods 22:49:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:49:32 INFO [loop_until]: OK (rc = 0) 22:49:32 DEBUG --- stdout --- 22:49:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12279m 8472Mi am-b59c765b7-bgsxp 10991m 8857Mi am-b59c765b7-ch5q4 11824m 8532Mi ds-cts-0 12196m 7860Mi ds-cts-1 10995m 7651Mi ds-cts-2 12629m 7937Mi ds-idrepo-0 5332m 3196Mi ds-idrepo-1 5192m 2838Mi ds-idrepo-2 5309m 2928Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 3m 1010Mi lodemon-7cfb6bd874-prp2f 1m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6529m 2004Mi 22:49:32 DEBUG --- stderr --- 22:49:32 DEBUG 22:49:32 INFO 22:49:32 INFO [loop_until]: kubectl --namespace=xlou top node 22:49:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390572" 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG --- http response --- 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390572" 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390572" 22:49:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390572" 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [loop_until]: OK (rc = 0) 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 DEBUG --- status code --- 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 DEBUG ----- output ----- 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 DEBUG --- status code --- 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG --- status code --- 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG --- stdout --- 22:49:32 DEBUG --- status code --- 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG { 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5426Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5263m 17% 6105Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11860m 39% 11112Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3001Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12314m 41% 10073Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 167m 0% 4980Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11915m 39% 11077Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 27012m 90% 2946Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11699m 39% 10690Mi 9% gke-xlou-cdm-ds-0f601751-j488 5600m 18% 5510Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11170m 37% 11177Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5831m 19% 5936Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12593m 42% 9609Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6738m 42% 5805Mi 9% 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 INFO [http_cmd]: http status code OK 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- stderr --- 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- status code --- 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG "data": { 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:32 DEBUG 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:49:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:49:32 DEBUG { 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG --- http response --- 22:49:32 DEBUG { 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG "result": [ 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG { 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG { 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG { 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG "data": { 22:49:32 DEBUG "data": { 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG { 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG { 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG ----- output ----- 22:49:32 DEBUG "data": { 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG { 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG { 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "data": { 22:49:32 DEBUG { 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "data": { 22:49:32 DEBUG "data": { 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "data": { 22:49:32 DEBUG { 22:49:32 DEBUG "data": { 22:49:32 DEBUG "data": { 22:49:32 DEBUG { 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "data": { 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "data": { 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "data": { 22:49:32 DEBUG "instance": "10.142.0.107:9100" 22:49:32 DEBUG "data": { 22:49:32 DEBUG "data": { 22:49:32 DEBUG "data": { 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "data": { 22:49:32 DEBUG { 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "data": { 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "data": { 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "status": "success", 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "data": { 22:49:32 DEBUG "data": { 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG }, 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "data": { 22:49:32 DEBUG { 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "data": { 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "data": { 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "data": { 22:49:32 DEBUG { 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG { 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "pod": "ds-idrepo-0" 22:49:32 DEBUG { 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "result": [ 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG "result": [ 22:49:32 DEBUG { 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG "resultType": "vector", 22:49:32 DEBUG "metric": { 22:49:32 DEBUG { 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "metric": { 22:49:32 DEBUG { 22:49:32 DEBUG 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:49:32 DEBUG { 22:49:32 DEBUG }, 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "result": [ 22:49:32 DEBUG { 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "metric": { 22:49:32 DEBUG { 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "result": [ 22:49:32 DEBUG "instance": "10.142.0.107:9100" 22:49:32 DEBUG "metric": { 22:49:32 DEBUG { 22:49:32 DEBUG { 22:49:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:49:32 DEBUG "metric": { 22:49:32 DEBUG 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "metric": { 22:49:32 DEBUG { 22:49:32 DEBUG }, 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:49:32 DEBUG { 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "instance": "10.142.0.107:9100" 22:49:32 DEBUG "pod": "ds-idrepo-0" 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "pod": "ds-idrepo-0" 22:49:32 DEBUG { 22:49:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:49:32 DEBUG "pod": "ds-idrepo-0" 22:49:32 DEBUG { 22:49:32 DEBUG }, 22:49:32 DEBUG "pod": "ds-idrepo-0" 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "metric": { 22:49:32 DEBUG }, 22:49:32 DEBUG "instance": "10.142.0.107:9100" 22:49:32 DEBUG 22:49:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:49:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:49:32 DEBUG "pod": "ds-cts-0" 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "pod": "ds-idrepo-0" 22:49:32 DEBUG 22:49:32 DEBUG }, 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "pod": "ds-idrepo-0" 22:49:32 DEBUG }, 22:49:32 DEBUG }, 22:49:32 DEBUG "pod": "ds-idrepo-0" 22:49:32 DEBUG }, 22:49:32 DEBUG "metric": { 22:49:32 DEBUG }, 22:49:32 DEBUG }, 22:49:32 DEBUG "metric": { 22:49:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:49:32 DEBUG }, 22:49:32 DEBUG "instance": "10.142.0.107:9100" 22:49:32 DEBUG "pod": "ds-cts-0" 22:49:32 DEBUG "value": [ 22:49:32 DEBUG }, 22:49:32 DEBUG [...] 22:49:32 DEBUG }, 22:49:32 DEBUG }, 22:49:32 DEBUG }, 22:49:32 DEBUG "pod": "ds-cts-0" 22:49:32 DEBUG 22:49:32 DEBUG }, 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:49:32 DEBUG }, 22:49:32 DEBUG "value": [ 22:49:32 DEBUG }, 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "value": [ 22:49:32 DEBUG }, 22:49:32 DEBUG "value": [ 22:49:32 DEBUG }, 22:49:32 DEBUG "value": [ 22:49:32 DEBUG }, 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "value": [ 22:49:32 DEBUG }, 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG }, 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG }, 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG "instance": "10.142.0.43:9100" 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG }, 22:49:32 DEBUG [...] 22:49:32 DEBUG [...] 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG "pod": "ds-idrepo-2" 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG [...] 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:49:32 DEBUG 22:49:32 DEBUG }, 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG "instance": "10.142.0.43:9100" 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG "instance": "10.142.0.43:9100" 22:49:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG "60842674.4" 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG }, 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG }, 22:49:32 DEBUG 22:49:32 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:49:32 DEBUG 22:49:32 DEBUG 22:49:32 DEBUG "pod": "ds-cts-2" 22:49:32 DEBUG 22:49:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:49:32 DEBUG 22:49:32 DEBUG "instance": "10.142.0.43:9100" 22:49:32 DEBUG }, 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG ] 22:49:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:49:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:49:32 DEBUG "pod": "ds-cts-2" 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG }, 22:49:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:49:32 DEBUG 22:49:32 DEBUG "pod": "ds-idrepo-2" 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "pod": "ds-idrepo-2" 22:49:32 DEBUG }, 22:49:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:49:32 DEBUG }, 22:49:32 DEBUG }, 22:49:32 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:49:32 DEBUG }, 22:49:32 DEBUG }, 22:49:32 DEBUG }, 22:49:32 DEBUG "1819.1333333333332" 22:49:32 DEBUG "instance": "10.142.0.43:9100" 22:49:32 DEBUG } 22:49:32 DEBUG 22:49:32 DEBUG }, 22:49:32 DEBUG }, 22:49:32 DEBUG }, 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG "1811.5" 22:49:32 DEBUG "0" 22:49:32 DEBUG "value": [ 22:49:32 DEBUG }, 22:49:32 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:49:32 DEBUG }, 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG }, 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:49:32 DEBUG "value": [ 22:49:32 DEBUG }, 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "value": [ 22:49:32 DEBUG ] 22:49:32 DEBUG }, 22:49:32 DEBUG ] 22:49:32 DEBUG 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG "pod": "ds-cts-2" 22:49:32 DEBUG }, 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG "value": [ 22:49:32 DEBUG }, 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "168629527552" 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG }, 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG } 22:49:32 DEBUG "value": [ 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG "0.00033333333333333327" 22:49:32 DEBUG }, 22:49:32 DEBUG "value": [ 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG "1819.2" 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG "0.016" 22:49:32 DEBUG "value": [ 22:49:32 DEBUG ] 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG "0" 22:49:32 DEBUG "6.605446672279158" 22:49:32 DEBUG "value": [ 22:49:32 DEBUG "0" 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG "0" 22:49:32 DEBUG "60845206.06666666" 22:49:32 DEBUG "0.15176666666666563" 22:49:32 DEBUG ] 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG "pod": "ds-cts-2" 22:49:32 DEBUG "1819.3" 22:49:32 DEBUG "328063.5" 22:49:32 DEBUG ] 22:49:32 DEBUG "value": [ 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG ] 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG } 22:49:32 DEBUG "0" 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG ] 22:49:32 DEBUG "2196557824" 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG "0.06999999999999995" 22:49:32 DEBUG }, 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG "0" 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG "0" 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG "0" 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG "value": [ 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG "0" 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG "0" 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG 1713390572, 22:49:32 DEBUG ] 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG "0" 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG ] 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:49:32 DEBUG } 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390632" 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:50:32 ERROR [http_cmd]: ERROR 22:50:32 ERROR --- status code --- 22:50:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:50:32 ERROR --- http response --- 22:50:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:50:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:50:32 INFO 22:50:32 INFO [loop_until]: kubectl --namespace=xlou top pods 22:50:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:50:32 INFO [loop_until]: OK (rc = 0) 22:50:32 DEBUG --- stdout --- 22:50:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11632m 8471Mi am-b59c765b7-bgsxp 11021m 8767Mi am-b59c765b7-ch5q4 11911m 8533Mi ds-cts-0 12633m 7841Mi ds-cts-1 11483m 7639Mi ds-cts-2 12864m 7925Mi ds-idrepo-0 5388m 3210Mi ds-idrepo-1 5172m 2839Mi ds-idrepo-2 5182m 2929Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 967Mi idm-b85f9d978-wnhxb 4m 1010Mi lodemon-7cfb6bd874-prp2f 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6709m 2006Mi 22:50:32 DEBUG --- stderr --- 22:50:32 DEBUG 22:50:32 INFO 22:50:32 INFO [loop_until]: kubectl --namespace=xlou top node 22:50:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390632" 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG ----- output ----- 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG "data": { 22:50:32 DEBUG "data": { 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:32 DEBUG "result": [ 22:50:32 DEBUG "result": [ 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:50:32 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:50:32 DEBUG "data": { 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG }, 22:50:32 DEBUG }, 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "result": [ 22:50:32 DEBUG 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:32 DEBUG 22:50:32 DEBUG { 22:50:32 DEBUG 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG 22:50:32 DEBUG "metric": { 22:50:32 DEBUG 22:50:32 DEBUG { 22:50:32 DEBUG 22:50:32 DEBUG "instance": "10.142.0.107:9100" 22:50:32 DEBUG [...] 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG [...] 22:50:32 DEBUG }, 22:50:32 DEBUG 22:50:32 DEBUG "data": { 22:50:32 DEBUG 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "result": [ 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:50:32 DEBUG { 22:50:32 DEBUG "pod": "idm-b85f9d978-txpwv" 22:50:32 DEBUG 22:50:32 DEBUG }, 22:50:32 DEBUG "metric": { 22:50:32 DEBUG }, 22:50:32 DEBUG [...] 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "instance": "10.142.0.107:9100" 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG }, 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG 22:50:32 DEBUG "1828.7" 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "0.0034811881014351993" 22:50:32 DEBUG 22:50:32 DEBUG ] 22:50:32 DEBUG 22:50:32 DEBUG ] 22:50:32 DEBUG "instance": "10.142.0.43:9100" 22:50:32 DEBUG } 22:50:32 DEBUG 22:50:32 DEBUG } 22:50:32 DEBUG }, 22:50:32 DEBUG ] 22:50:32 DEBUG 22:50:32 DEBUG ] 22:50:32 DEBUG "value": [ 22:50:32 DEBUG } 22:50:32 DEBUG [...] 22:50:32 DEBUG } 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG } 22:50:32 DEBUG 22:50:32 DEBUG } 22:50:32 DEBUG "0.10006666666666698" 22:50:32 DEBUG 22:50:32 DEBUG ] 22:50:32 DEBUG 22:50:32 DEBUG } 22:50:32 DEBUG "instance": "10.142.0.43:9100" 22:50:32 DEBUG ] 22:50:32 DEBUG }, 22:50:32 DEBUG } 22:50:32 DEBUG "value": [ 22:50:32 DEBUG } 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG "60828262.599999994" 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390632" 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 DEBUG --- http response --- 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [loop_until]: OK (rc = 0) 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390632" 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 DEBUG --- stdout --- 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG ----- output ----- 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5436Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5292m 17% 6133Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11812m 39% 11111Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 2998Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12420m 41% 10076Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 164m 0% 4979Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11979m 40% 11080Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 28070m 94% 2951Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11812m 39% 10613Mi 9% gke-xlou-cdm-ds-0f601751-j488 5614m 18% 5508Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11095m 37% 11183Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5809m 19% 5933Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12754m 42% 9611Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6857m 43% 5806Mi 9% 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG { 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG --- stderr --- 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG --- http response --- 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 INFO [http_cmd]: http status code OK 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:32 DEBUG "data": { 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG --- status code --- 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG { 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG { 22:50:32 DEBUG "result": [ 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG { 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG { 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG --- http response --- 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG { 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG "data": { 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG { 22:50:32 DEBUG "data": { 22:50:32 DEBUG { 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:50:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "data": { 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "data": { 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG ----- output ----- 22:50:32 DEBUG "data": { 22:50:32 DEBUG "data": { 22:50:32 DEBUG "data": { 22:50:32 DEBUG "data": { 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "data": { 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "data": { 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "data": { 22:50:32 DEBUG "data": { 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "instance": "10.142.0.107:9100" 22:50:32 DEBUG "data": { 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "data": { 22:50:32 DEBUG "result": [ 22:50:32 DEBUG "data": { 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "data": { 22:50:32 DEBUG "data": { 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "result": [ 22:50:32 DEBUG "result": [ 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "result": [ 22:50:32 DEBUG }, 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "result": [ 22:50:32 DEBUG { 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "status": "success", 22:50:32 DEBUG "result": [ 22:50:32 DEBUG { 22:50:32 DEBUG "result": [ 22:50:32 DEBUG "result": [ 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "result": [ 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG "data": { 22:50:32 DEBUG "result": [ 22:50:32 DEBUG "result": [ 22:50:32 DEBUG { 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "result": [ 22:50:32 DEBUG { 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "instance": "10.142.0.107:9100" 22:50:32 DEBUG "result": [ 22:50:32 DEBUG "data": { 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "result": [ 22:50:32 DEBUG "metric": { 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG "result": [ 22:50:32 DEBUG "result": [ 22:50:32 DEBUG { 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG "metric": { 22:50:32 DEBUG 22:50:32 DEBUG { 22:50:32 DEBUG "pod": "ds-idrepo-0" 22:50:32 DEBUG }, 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG { 22:50:32 DEBUG "data": { 22:50:32 DEBUG "result": [ 22:50:32 DEBUG { 22:50:32 DEBUG "pod": "ds-cts-0" 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "metric": { 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:50:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:50:32 DEBUG "result": [ 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:50:32 DEBUG 22:50:32 DEBUG "metric": { 22:50:32 DEBUG }, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "result": [ 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:50:32 DEBUG "resultType": "vector", 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "pod": "ds-idrepo-0" 22:50:32 DEBUG }, 22:50:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "pod": "ds-idrepo-0" 22:50:32 DEBUG }, 22:50:32 DEBUG }, 22:50:32 DEBUG { 22:50:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:50:32 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:50:32 DEBUG }, 22:50:32 DEBUG 22:50:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 22:50:32 DEBUG { 22:50:32 DEBUG { 22:50:32 DEBUG }, 22:50:32 DEBUG "result": [ 22:50:32 DEBUG "value": [ 22:50:32 DEBUG }, 22:50:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:50:32 DEBUG }, 22:50:32 DEBUG "pod": "ds-idrepo-0" 22:50:32 DEBUG "pod": "ds-cts-0" 22:50:32 DEBUG }, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "metric": { 22:50:32 DEBUG }, 22:50:32 DEBUG }, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG [...] 22:50:32 DEBUG }, 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "metric": { 22:50:32 DEBUG "value": [ 22:50:32 DEBUG { 22:50:32 DEBUG 22:50:32 DEBUG "value": [ 22:50:32 DEBUG }, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG }, 22:50:32 DEBUG }, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "instance": "10.142.0.107:9100" 22:50:32 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:50:32 DEBUG 22:50:32 DEBUG "metric": { 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG }, 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG [...] 22:50:32 DEBUG }, 22:50:32 DEBUG }, 22:50:32 DEBUG 22:50:32 DEBUG "pod": "ds-cts-0" 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG [...] 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG [...] 22:50:32 DEBUG 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 22:50:32 DEBUG }, 22:50:32 DEBUG [...] 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG [...] 22:50:32 DEBUG 22:50:32 DEBUG "value": [ 22:50:32 DEBUG [...] 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG [...] 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 22:50:32 DEBUG [...] 22:50:32 DEBUG 22:50:32 DEBUG [...] 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG [...] 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG [...] 22:50:32 DEBUG [...] 22:50:32 DEBUG "instance": "10.142.0.43:9100" 22:50:32 DEBUG [...] 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG [...] 22:50:32 DEBUG 22:50:32 DEBUG [...] 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG }, 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "instance": "10.142.0.43:9100" 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG [...] 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG }, 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 22:50:32 DEBUG "pod": "ds-idrepo-2" 22:50:32 DEBUG }, 22:50:32 DEBUG [...] 22:50:32 DEBUG [...] 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "pod": "ds-idrepo-2" 22:50:32 DEBUG "pod": "ds-cts-2" 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:50:32 DEBUG "value": [ 22:50:32 DEBUG [...] 22:50:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG 22:50:32 DEBUG }, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:50:32 DEBUG [...] 22:50:32 DEBUG }, 22:50:32 DEBUG }, 22:50:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG "pod": "ds-cts-2" 22:50:32 DEBUG }, 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG 22:50:32 DEBUG }, 22:50:32 DEBUG "pod": "idm-b85f9d978-txpwv" 22:50:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:50:32 DEBUG "60825984.3" 22:50:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG }, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "value": [ 22:50:32 DEBUG }, 22:50:32 DEBUG "pod": "ds-idrepo-2" 22:50:32 DEBUG 22:50:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:50:32 DEBUG }, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "2197688320" 22:50:32 DEBUG 22:50:32 DEBUG "value": [ 22:50:32 DEBUG }, 22:50:32 DEBUG }, 22:50:32 DEBUG ] 22:50:32 DEBUG }, 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG "171213976576" 22:50:32 DEBUG 22:50:32 DEBUG 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG }, 22:50:32 DEBUG "pod": "ds-cts-2" 22:50:32 DEBUG }, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG ] 22:50:32 DEBUG 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "value": [ 22:50:32 DEBUG } 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "0" 22:50:32 DEBUG ] 22:50:32 DEBUG "instance": "10.142.0.43:9100" 22:50:32 DEBUG "pod": "idm-b85f9d978-txpwv" 22:50:32 DEBUG "0.016" 22:50:32 DEBUG 22:50:32 DEBUG "0" 22:50:32 DEBUG "0.000033333333333333335" 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG }, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG "6.570264370137473" 22:50:32 DEBUG } 22:50:32 DEBUG "pod": "idm-b85f9d978-txpwv" 22:50:32 DEBUG "1822.7333333333333" 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG ] 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG }, 22:50:32 DEBUG }, 22:50:32 DEBUG ] 22:50:32 DEBUG 22:50:32 DEBUG ] 22:50:32 DEBUG ] 22:50:32 DEBUG "327888" 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG "value": [ 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG "1829.4333333333334" 22:50:32 DEBUG ] 22:50:32 DEBUG ] 22:50:32 DEBUG }, 22:50:32 DEBUG ] 22:50:32 DEBUG "0" 22:50:32 DEBUG "1828.9666666666667" 22:50:32 DEBUG } 22:50:32 DEBUG "0" 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG "value": [ 22:50:32 DEBUG "value": [ 22:50:32 DEBUG } 22:50:32 DEBUG "pod": "ds-cts-2" 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG "0" 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG ] 22:50:32 DEBUG "0" 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG "value": [ 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG ] 22:50:32 DEBUG }, 22:50:32 DEBUG ] 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG "0" 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG "0.052333333333333204" 22:50:32 DEBUG "0" 22:50:32 DEBUG } 22:50:32 DEBUG "value": [ 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG "9771.13109323237" 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG 1713390632, 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG "0" 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG ] 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG ] 22:50:32 DEBUG } 22:50:32 DEBUG } 22:50:32 DEBUG } 22:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390692" 22:51:32 ERROR [http_cmd]: ERROR 22:51:32 ERROR --- status code --- 22:51:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:51:32 ERROR --- http response --- 22:51:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:51:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390692" 22:51:32 INFO [http_cmd]: http status code OK 22:51:32 DEBUG --- status code --- 22:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:32 DEBUG --- http response --- 22:51:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:51:32 INFO 22:51:32 INFO [loop_until]: kubectl --namespace=xlou top pods 22:51:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:51:32 INFO [loop_until]: OK (rc = 0) 22:51:32 DEBUG --- stdout --- 22:51:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11610m 8909Mi am-b59c765b7-bgsxp 11119m 8767Mi am-b59c765b7-ch5q4 11921m 8534Mi ds-cts-0 12319m 7864Mi ds-cts-1 11784m 7669Mi ds-cts-2 12855m 7920Mi ds-idrepo-0 5324m 3210Mi ds-idrepo-1 5279m 2838Mi ds-idrepo-2 5262m 2930Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1009Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6545m 2046Mi 22:51:32 DEBUG --- stderr --- 22:51:32 DEBUG 22:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390692" 22:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390692" 22:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390692" 22:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390692" 22:51:32 INFO [http_cmd]: http status code OK 22:51:32 INFO [http_cmd]: http status code OK 22:51:32 DEBUG --- status code --- 22:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:32 DEBUG --- http response --- 22:51:32 DEBUG --- status code --- 22:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:32 DEBUG ----- output ----- 22:51:32 DEBUG --- http response --- 22:51:32 DEBUG { 22:51:32 DEBUG "status": "success", 22:51:32 DEBUG "data": { 22:51:32 DEBUG "resultType": "vector", 22:51:32 DEBUG "result": [ 22:51:32 DEBUG { 22:51:32 DEBUG "metric": { 22:51:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:51:32 DEBUG }, 22:51:32 DEBUG "value": [ 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG [...] 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:32 DEBUG }, 22:51:32 INFO [http_cmd]: http status code OK 22:51:32 INFO [http_cmd]: http status code OK 22:51:32 DEBUG ----- output ----- 22:51:32 DEBUG "value": [ 22:51:32 DEBUG --- status code --- 22:51:32 DEBUG --- status code --- 22:51:32 DEBUG { 22:51:32 DEBUG 1713390692, 22:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:32 DEBUG "status": "success", 22:51:32 DEBUG "1829.0666666666666" 22:51:32 DEBUG --- http response --- 22:51:32 DEBUG --- http response --- 22:51:32 DEBUG "data": { 22:51:32 DEBUG ] 22:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:32 DEBUG "resultType": "vector", 22:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:32 DEBUG } 22:51:32 DEBUG ----- output ----- 22:51:32 DEBUG "result": [ 22:51:32 DEBUG ----- output ----- 22:51:32 DEBUG ] 22:51:32 DEBUG { 22:51:32 DEBUG { 22:51:32 DEBUG { 22:51:32 DEBUG } 22:51:32 DEBUG "status": "success", 22:51:32 DEBUG "metric": { 22:51:32 DEBUG "status": "success", 22:51:32 DEBUG } 22:51:32 DEBUG "data": { 22:51:32 DEBUG "instance": "10.142.0.107:9100" 22:51:32 DEBUG "data": { 22:51:32 DEBUG "resultType": "vector", 22:51:32 DEBUG }, 22:51:32 DEBUG "resultType": "vector", 22:51:32 DEBUG "result": [ 22:51:32 DEBUG "value": [ 22:51:32 DEBUG "result": [ 22:51:32 DEBUG { 22:51:32 DEBUG 22:51:32 DEBUG { 22:51:32 DEBUG "metric": { 22:51:32 DEBUG 22:51:32 DEBUG "metric": { 22:51:32 DEBUG "pod": "ds-idrepo-0" 22:51:32 DEBUG 22:51:32 DEBUG "instance": "10.142.0.107:9100" 22:51:32 DEBUG }, 22:51:32 DEBUG [...] 22:51:32 DEBUG }, 22:51:32 DEBUG "value": [ 22:51:32 DEBUG 22:51:32 DEBUG "value": [ 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG "instance": "10.142.0.43:9100" 22:51:32 DEBUG 22:51:32 DEBUG [...] 22:51:32 DEBUG }, 22:51:32 DEBUG "value": [ 22:51:32 DEBUG 22:51:32 DEBUG 1713390692, 22:51:32 DEBUG 22:51:32 DEBUG "0.1698666666666668" 22:51:32 DEBUG 22:51:32 DEBUG ] 22:51:32 DEBUG "pod": "am-b59c765b7-7xg4f" 22:51:32 DEBUG } 22:51:32 DEBUG }, 22:51:32 DEBUG [...] 22:51:32 DEBUG ] 22:51:32 DEBUG "value": [ 22:51:32 DEBUG 22:51:32 DEBUG } 22:51:32 DEBUG 1713390692, 22:51:32 DEBUG 22:51:32 DEBUG } 22:51:32 DEBUG "11.797846233436779" 22:51:32 DEBUG 22:51:32 DEBUG ] 22:51:32 DEBUG "instance": "10.142.0.43:9100" 22:51:32 DEBUG } 22:51:32 DEBUG }, 22:51:32 DEBUG ] 22:51:32 DEBUG "value": [ 22:51:32 DEBUG } 22:51:32 DEBUG 1713390692, 22:51:32 DEBUG } 22:51:32 DEBUG "60641931.96666667" 22:51:32 DEBUG ] 22:51:32 DEBUG } 22:51:32 DEBUG ] 22:51:32 DEBUG } 22:51:32 DEBUG } 22:51:32 INFO 22:51:32 INFO [loop_until]: kubectl --namespace=xlou top node 22:51:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:51:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390692" 22:51:32 INFO [http_cmd]: http status code OK 22:51:32 DEBUG --- status code --- 22:51:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:32 DEBUG --- http response --- 22:51:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:32 DEBUG ----- output ----- 22:51:32 DEBUG { 22:51:32 DEBUG "status": "success", 22:51:32 DEBUG "data": { 22:51:32 DEBUG "resultType": "vector", 22:51:32 DEBUG "result": [ 22:51:32 DEBUG { 22:51:32 DEBUG "metric": { 22:51:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:51:32 DEBUG }, 22:51:32 DEBUG "value": [ 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG [...] 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG 22:51:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:51:32 DEBUG }, 22:51:32 DEBUG "value": [ 22:51:32 DEBUG 1713390692, 22:51:32 DEBUG "2238439424" 22:51:32 DEBUG ] 22:51:32 DEBUG } 22:51:32 DEBUG ] 22:51:32 DEBUG } 22:51:32 DEBUG } 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390693" 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390693" 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 INFO [loop_until]: OK (rc = 0) 22:51:33 DEBUG --- stdout --- 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 DEBUG --- status code --- 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 DEBUG --- status code --- 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 DEBUG --- status code --- 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5440Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5271m 17% 6126Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11808m 39% 11111Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3009Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11966m 40% 10080Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 4971Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11893m 39% 11525Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27773m 93% 2912Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11834m 39% 10611Mi 9% gke-xlou-cdm-ds-0f601751-j488 5596m 18% 5508Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11571m 38% 11203Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5919m 19% 5949Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13041m 43% 9607Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6709m 42% 5842Mi 9% 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG --- status code --- 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG --- stderr --- 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:33 DEBUG --- http response --- 22:51:33 INFO [http_cmd]: http status code OK 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:33 DEBUG --- status code --- 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG { 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG { 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:51:33 DEBUG { 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG { 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG { 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG { 22:51:33 DEBUG { 22:51:33 DEBUG { 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:33 DEBUG { 22:51:33 DEBUG { 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG { 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG { 22:51:33 DEBUG --- http response --- 22:51:33 DEBUG { 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG { 22:51:33 DEBUG { 22:51:33 DEBUG { 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG "data": { 22:51:33 DEBUG { 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG "data": { 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG { 22:51:33 DEBUG "data": { 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG "data": { 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG "data": { 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG "data": { 22:51:33 DEBUG "data": { 22:51:33 DEBUG "data": { 22:51:33 DEBUG { 22:51:33 DEBUG "data": { 22:51:33 DEBUG "data": { 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "data": { 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "data": { 22:51:33 DEBUG ----- output ----- 22:51:33 DEBUG "data": { 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "data": { 22:51:33 DEBUG "data": { 22:51:33 DEBUG "data": { 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "data": { 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "data": { 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG { 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "result": [ 22:51:33 DEBUG { 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "data": { 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "result": [ 22:51:33 DEBUG { 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG { 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "status": "success", 22:51:33 DEBUG "result": [ 22:51:33 DEBUG { 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "result": [ 22:51:33 DEBUG { 22:51:33 DEBUG "metric": { 22:51:33 DEBUG "result": [ 22:51:33 DEBUG { 22:51:33 DEBUG { 22:51:33 DEBUG "metric": { 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG { 22:51:33 DEBUG { 22:51:33 DEBUG "metric": { 22:51:33 DEBUG { 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "metric": { 22:51:33 DEBUG { 22:51:33 DEBUG "data": { 22:51:33 DEBUG { 22:51:33 DEBUG "metric": { 22:51:33 DEBUG { 22:51:33 DEBUG { 22:51:33 DEBUG { 22:51:33 DEBUG "metric": { 22:51:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:51:33 DEBUG { 22:51:33 DEBUG "metric": { 22:51:33 DEBUG { 22:51:33 DEBUG "pod": "ds-idrepo-0" 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "metric": { 22:51:33 DEBUG "metric": { 22:51:33 DEBUG "instance": "10.142.0.107:9100" 22:51:33 DEBUG "metric": { 22:51:33 DEBUG { 22:51:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:51:33 DEBUG "metric": { 22:51:33 DEBUG "resultType": "vector", 22:51:33 DEBUG "metric": { 22:51:33 DEBUG "instance": "10.142.0.107:9100" 22:51:33 DEBUG "metric": { 22:51:33 DEBUG "metric": { 22:51:33 DEBUG "pod": "ds-idrepo-0" 22:51:33 DEBUG "metric": { 22:51:33 DEBUG }, 22:51:33 DEBUG "metric": { 22:51:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:51:33 DEBUG }, 22:51:33 DEBUG }, 22:51:33 DEBUG { 22:51:33 DEBUG "pod": "ds-idrepo-0" 22:51:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:51:33 DEBUG }, 22:51:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:51:33 DEBUG "metric": { 22:51:33 DEBUG }, 22:51:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:51:33 DEBUG "result": [ 22:51:33 DEBUG "pod": "ds-cts-0" 22:51:33 DEBUG }, 22:51:33 DEBUG "pod": "ds-idrepo-0" 22:51:33 DEBUG "pod": "ds-cts-0" 22:51:33 DEBUG }, 22:51:33 DEBUG "pod": "ds-idrepo-0" 22:51:33 DEBUG "value": [ 22:51:33 DEBUG "pod": "ds-idrepo-0" 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 22:51:33 DEBUG "value": [ 22:51:33 DEBUG "metric": { 22:51:33 DEBUG }, 22:51:33 DEBUG }, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG }, 22:51:33 DEBUG "pod": "ds-idrepo-0" 22:51:33 DEBUG "value": [ 22:51:33 DEBUG }, 22:51:33 DEBUG { 22:51:33 DEBUG }, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG }, 22:51:33 DEBUG }, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG }, 22:51:33 DEBUG 22:51:33 DEBUG }, 22:51:33 DEBUG 22:51:33 DEBUG "metric": { 22:51:33 DEBUG 22:51:33 DEBUG "instance": "10.142.0.107:9100" 22:51:33 DEBUG "value": [ 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 22:51:33 DEBUG "value": [ 22:51:33 DEBUG }, 22:51:33 DEBUG 22:51:33 DEBUG "value": [ 22:51:33 DEBUG "metric": { 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 22:51:33 DEBUG "value": [ 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:51:33 DEBUG 22:51:33 DEBUG }, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG }, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG "pod": "ds-cts-0" 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG }, 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG [...] 22:51:33 DEBUG [...] 22:51:33 DEBUG 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG [...] 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG }, 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG [...] 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG 22:51:33 DEBUG }, 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG "value": [ 22:51:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:51:33 DEBUG "instance": "10.142.0.43:9100" 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG "value": [ 22:51:33 DEBUG "pod": "ds-cts-2" 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG "pod": "ds-idrepo-2" 22:51:33 DEBUG 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG }, 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG "pod": "am-b59c765b7-7xg4f" 22:51:33 DEBUG 22:51:33 DEBUG }, 22:51:33 DEBUG }, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG [...] 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG }, 22:51:33 DEBUG "instance": "10.142.0.43:9100" 22:51:33 DEBUG "pod": "ds-cts-2" 22:51:33 DEBUG "pod": "ds-cts-2" 22:51:33 DEBUG }, 22:51:33 DEBUG "pod": "ds-idrepo-2" 22:51:33 DEBUG "1822.6333333333332" 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG "instance": "10.142.0.43:9100" 22:51:33 DEBUG }, 22:51:33 DEBUG "pod": "ds-idrepo-2" 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG "pod": "am-b59c765b7-7xg4f" 22:51:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:51:33 DEBUG 22:51:33 DEBUG 22:51:33 DEBUG "0.016" 22:51:33 DEBUG "value": [ 22:51:33 DEBUG }, 22:51:33 DEBUG }, 22:51:33 DEBUG }, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG ] 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG "pod": "am-b59c765b7-7xg4f" 22:51:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:51:33 DEBUG }, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG }, 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG "60640118.83333333" 22:51:33 DEBUG }, 22:51:33 DEBUG }, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 22:51:33 DEBUG ] 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG }, 22:51:33 DEBUG "0" 22:51:33 DEBUG } 22:51:33 DEBUG "1828.7" 22:51:33 DEBUG ] 22:51:33 DEBUG }, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG "6.56068251984697" 22:51:33 DEBUG ] 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG 22:51:33 DEBUG } 22:51:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:51:33 DEBUG "0" 22:51:33 DEBUG "174166950912" 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG }, 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG "0" 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG "0" 22:51:33 DEBUG "pod": "ds-cts-2" 22:51:33 DEBUG ] 22:51:33 DEBUG }, 22:51:33 DEBUG ] 22:51:33 DEBUG ] 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG "0.00002210444444444444" 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG ] 22:51:33 DEBUG ] 22:51:33 DEBUG ] 22:51:33 DEBUG "value": [ 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG "0.08466666666666701" 22:51:33 DEBUG ] 22:51:33 DEBUG "0" 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG "1093.6884616668335" 22:51:33 DEBUG ] 22:51:33 DEBUG }, 22:51:33 DEBUG } 22:51:33 DEBUG "value": [ 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG "0" 22:51:33 DEBUG ] 22:51:33 DEBUG "0" 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG "328149.5" 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG "value": [ 22:51:33 DEBUG } 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG ] 22:51:33 DEBUG ] 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG "8476.08557791796" 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG 1713390693, 22:51:33 DEBUG "1830.3666666666666" 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG "0" 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG ] 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:51:33 DEBUG } 22:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390752" 22:52:32 ERROR [http_cmd]: ERROR 22:52:32 ERROR --- status code --- 22:52:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:52:32 ERROR --- http response --- 22:52:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:52:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390752" 22:52:32 INFO [http_cmd]: http status code OK 22:52:32 DEBUG --- status code --- 22:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:32 DEBUG --- http response --- 22:52:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:52:32 INFO 22:52:32 INFO [loop_until]: kubectl --namespace=xlou top pods 22:52:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:52:32 INFO [loop_until]: OK (rc = 0) 22:52:32 DEBUG --- stdout --- 22:52:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12050m 8916Mi am-b59c765b7-bgsxp 11076m 8769Mi am-b59c765b7-ch5q4 11697m 8534Mi ds-cts-0 12835m 7839Mi ds-cts-1 11621m 7652Mi ds-cts-2 12959m 7935Mi ds-idrepo-0 5344m 3210Mi ds-idrepo-1 5223m 2838Mi ds-idrepo-2 5267m 2937Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 967Mi idm-b85f9d978-wnhxb 3m 1009Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6606m 2049Mi 22:52:32 DEBUG --- stderr --- 22:52:32 DEBUG 22:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390752" 22:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390752" 22:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390752" 22:52:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390752" 22:52:32 INFO [http_cmd]: http status code OK 22:52:32 INFO [http_cmd]: http status code OK 22:52:32 DEBUG --- status code --- 22:52:32 DEBUG --- status code --- 22:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:32 DEBUG --- http response --- 22:52:32 DEBUG --- http response --- 22:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:32 DEBUG ----- output ----- 22:52:32 DEBUG { 22:52:32 DEBUG "status": "success", 22:52:32 DEBUG "data": { 22:52:32 DEBUG "resultType": "vector", 22:52:32 DEBUG "result": [ 22:52:32 DEBUG { 22:52:32 DEBUG "metric": { 22:52:32 DEBUG "pod": "am-b59c765b7-ch5q4" 22:52:32 DEBUG }, 22:52:32 DEBUG "value": [ 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG [...] 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG "pod": "am-b59c765b7-bgsxp" 22:52:32 DEBUG }, 22:52:32 DEBUG "value": [ 22:52:32 DEBUG 1713390752, 22:52:32 DEBUG "1821.1" 22:52:32 DEBUG ] 22:52:32 DEBUG } 22:52:32 DEBUG ] 22:52:32 DEBUG } 22:52:32 DEBUG } 22:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:32 DEBUG ----- output ----- 22:52:32 DEBUG { 22:52:32 DEBUG "status": "success", 22:52:32 DEBUG "data": { 22:52:32 DEBUG "resultType": "vector", 22:52:32 DEBUG "result": [ 22:52:32 DEBUG { 22:52:32 DEBUG "metric": { 22:52:32 DEBUG "instance": "10.142.0.107:9100" 22:52:32 DEBUG }, 22:52:32 DEBUG "value": [ 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG [...] 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG "instance": "10.142.0.43:9100" 22:52:32 DEBUG }, 22:52:32 DEBUG "value": [ 22:52:32 DEBUG 1713390752, 22:52:32 DEBUG "0.1593999999999994" 22:52:32 DEBUG ] 22:52:32 DEBUG } 22:52:32 DEBUG ] 22:52:32 DEBUG } 22:52:32 DEBUG } 22:52:32 INFO [http_cmd]: http status code OK 22:52:32 INFO [http_cmd]: http status code OK 22:52:32 DEBUG --- status code --- 22:52:32 DEBUG --- status code --- 22:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:32 DEBUG --- http response --- 22:52:32 DEBUG --- http response --- 22:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:52:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:32 DEBUG ----- output ----- 22:52:32 DEBUG ----- output ----- 22:52:32 DEBUG { 22:52:32 DEBUG { 22:52:32 DEBUG "status": "success", 22:52:32 DEBUG "status": "success", 22:52:32 DEBUG "data": { 22:52:32 DEBUG "data": { 22:52:32 DEBUG "resultType": "vector", 22:52:32 DEBUG "resultType": "vector", 22:52:32 DEBUG "result": [ 22:52:32 DEBUG "result": [ 22:52:32 DEBUG { 22:52:32 DEBUG { 22:52:32 DEBUG "metric": { 22:52:32 DEBUG "metric": { 22:52:32 DEBUG "pod": "ds-idrepo-1" 22:52:32 DEBUG "instance": "10.142.0.107:9100" 22:52:32 DEBUG }, 22:52:32 DEBUG }, 22:52:32 DEBUG "value": [ 22:52:32 DEBUG "value": [ 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG [...] 22:52:32 DEBUG [...] 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG 22:52:32 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 22:52:32 DEBUG "instance": "10.142.0.43:9100" 22:52:32 DEBUG }, 22:52:32 DEBUG }, 22:52:32 DEBUG "value": [ 22:52:32 DEBUG "value": [ 22:52:32 DEBUG 1713390752, 22:52:32 DEBUG 1713390752, 22:52:32 DEBUG "6.580785101750548" 22:52:32 DEBUG "60851364.16666667" 22:52:32 DEBUG ] 22:52:32 DEBUG ] 22:52:32 DEBUG } 22:52:32 DEBUG } 22:52:32 DEBUG ] 22:52:32 DEBUG ] 22:52:32 DEBUG } 22:52:32 DEBUG } 22:52:32 DEBUG } 22:52:32 DEBUG } 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390753" 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG { 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "data": { 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "result": [ 22:52:33 DEBUG { 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:52:33 DEBUG }, 22:52:33 DEBUG "value": [ 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG [...] 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:52:33 DEBUG }, 22:52:33 DEBUG "value": [ 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG "2240737280" 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 INFO 22:52:33 INFO [loop_until]: kubectl --namespace=xlou top node 22:52:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390753" 22:52:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390753" 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 INFO [loop_until]: OK (rc = 0) 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 DEBUG --- status code --- 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 DEBUG --- status code --- 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG --- stdout --- 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG --- status code --- 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 5438Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5273m 17% 6139Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11876m 39% 11109Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 2992Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12479m 41% 10081Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 4968Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12039m 40% 11524Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28159m 94% 2924Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11878m 39% 10612Mi 9% gke-xlou-cdm-ds-0f601751-j488 5564m 18% 5517Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11579m 38% 11211Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5898m 19% 5942Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13026m 43% 9615Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6775m 42% 5852Mi 9% 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG --- stderr --- 22:52:33 DEBUG --- http response --- 22:52:33 INFO [http_cmd]: http status code OK 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG 22:52:33 DEBUG --- status code --- 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:52:33 DEBUG { 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG { 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG { 22:52:33 DEBUG --- http response --- 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG { 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:33 DEBUG { 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "data": { 22:52:33 DEBUG { 22:52:33 DEBUG "data": { 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "data": { 22:52:33 DEBUG { 22:52:33 DEBUG "data": { 22:52:33 DEBUG "data": { 22:52:33 DEBUG "data": { 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG "data": { 22:52:33 DEBUG ----- output ----- 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "data": { 22:52:33 DEBUG "data": { 22:52:33 DEBUG "data": { 22:52:33 DEBUG "data": { 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "data": { 22:52:33 DEBUG "data": { 22:52:33 DEBUG "data": { 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "data": { 22:52:33 DEBUG { 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG { 22:52:33 DEBUG "data": { 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "data": { 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "data": { 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "status": "success", 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "result": [ 22:52:33 DEBUG { 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "result": [ 22:52:33 DEBUG { 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "data": { 22:52:33 DEBUG { 22:52:33 DEBUG "data": { 22:52:33 DEBUG "result": [ 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "metric": { 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG { 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "metric": { 22:52:33 DEBUG { 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG { 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "metric": { 22:52:33 DEBUG { 22:52:33 DEBUG "pod": "ds-idrepo-0" 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:52:33 DEBUG { 22:52:33 DEBUG "pod": "ds-cts-0" 22:52:33 DEBUG "pod": "ds-idrepo-0" 22:52:33 DEBUG "instance": "10.142.0.107:9100" 22:52:33 DEBUG "resultType": "vector", 22:52:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:52:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:52:33 DEBUG "result": [ 22:52:33 DEBUG "metric": { 22:52:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:52:33 DEBUG "pod": "ds-cts-0" 22:52:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:52:33 DEBUG "instance": "10.142.0.107:9100" 22:52:33 DEBUG "pod": "ds-idrepo-1" 22:52:33 DEBUG "metric": { 22:52:33 DEBUG }, 22:52:33 DEBUG "pod": "ds-idrepo-1" 22:52:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:52:33 DEBUG "pod": "ds-idrepo-1" 22:52:33 DEBUG }, 22:52:33 DEBUG "metric": { 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG "result": [ 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG { 22:52:33 DEBUG "pod": "ds-cts-0" 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:52:33 DEBUG "value": [ 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "pod": "ds-idrepo-0" 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG { 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "metric": { 22:52:33 DEBUG }, 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG }, 22:52:33 DEBUG 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG 22:52:33 DEBUG }, 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG "metric": { 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG "pod": "ds-idrepo-0" 22:52:33 DEBUG "value": [ 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG "value": [ 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG [...] 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG "value": [ 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG "instance": "10.142.0.107:9100" 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG }, 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG [...] 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG "value": [ 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG [...] 22:52:33 DEBUG 22:52:33 DEBUG [...] 22:52:33 DEBUG [...] 22:52:33 DEBUG 22:52:33 DEBUG }, 22:52:33 DEBUG [...] 22:52:33 DEBUG [...] 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG [...] 22:52:33 DEBUG [...] 22:52:33 DEBUG [...] 22:52:33 DEBUG [...] 22:52:33 DEBUG [...] 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG [...] 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG "value": [ 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG [...] 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG [...] 22:52:33 DEBUG [...] 22:52:33 DEBUG "pod": "ds-idrepo-2" 22:52:33 DEBUG [...] 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG [...] 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG }, 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG "instance": "10.142.0.43:9100" 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG "value": [ 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:52:33 DEBUG 22:52:33 DEBUG "pod": "ds-cts-2" 22:52:33 DEBUG "pod": "ds-idrepo-2" 22:52:33 DEBUG }, 22:52:33 DEBUG 22:52:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:52:33 DEBUG }, 22:52:33 DEBUG [...] 22:52:33 DEBUG 22:52:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:52:33 DEBUG "pod": "ds-cts-2" 22:52:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:52:33 DEBUG "instance": "10.142.0.43:9100" 22:52:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:52:33 DEBUG 22:52:33 DEBUG 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG 22:52:33 DEBUG }, 22:52:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:52:33 DEBUG 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG "value": [ 22:52:33 DEBUG [...] 22:52:33 DEBUG }, 22:52:33 DEBUG "value": [ 22:52:33 DEBUG 22:52:33 DEBUG "pod": "ds-cts-2" 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG }, 22:52:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:52:33 DEBUG }, 22:52:33 DEBUG "0" 22:52:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:52:33 DEBUG "value": [ 22:52:33 DEBUG }, 22:52:33 DEBUG "pod": "ds-cts-2" 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG 22:52:33 DEBUG "value": [ 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG 22:52:33 DEBUG }, 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:52:33 DEBUG ] 22:52:33 DEBUG }, 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG "value": [ 22:52:33 DEBUG }, 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG "60849194.46666667" 22:52:33 DEBUG 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG "6.682895264308436" 22:52:33 DEBUG 22:52:33 DEBUG "value": [ 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG }, 22:52:33 DEBUG } 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "0" 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG "value": [ 22:52:33 DEBUG "0" 22:52:33 DEBUG ] 22:52:33 DEBUG "0" 22:52:33 DEBUG 22:52:33 DEBUG "1820.5" 22:52:33 DEBUG ] 22:52:33 DEBUG "pod": "ds-idrepo-2" 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG "0" 22:52:33 DEBUG "0" 22:52:33 DEBUG "1821.1" 22:52:33 DEBUG "177513841664" 22:52:33 DEBUG "0" 22:52:33 DEBUG "1820.0333333333333" 22:52:33 DEBUG "value": [ 22:52:33 DEBUG ] 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG ] 22:52:33 DEBUG "0.016" 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG "instance": "10.142.0.43:9100" 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG }, 22:52:33 DEBUG "0" 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG } 22:52:33 DEBUG "0" 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG "0" 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG }, 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG "328351.5" 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG "value": [ 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG "value": [ 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG 1713390753, 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG "0.07766666666666659" 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG "0" 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG ] 22:52:33 DEBUG ] 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:52:33 DEBUG } 22:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390812" 22:53:32 ERROR [http_cmd]: ERROR 22:53:32 ERROR --- status code --- 22:53:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:53:32 ERROR --- http response --- 22:53:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:53:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:53:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390812" 22:53:32 INFO [http_cmd]: http status code OK 22:53:32 DEBUG --- status code --- 22:53:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:32 DEBUG --- http response --- 22:53:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:53:32 INFO 22:53:32 INFO [loop_until]: kubectl --namespace=xlou top pods 22:53:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:53:32 INFO [loop_until]: OK (rc = 0) 22:53:32 DEBUG --- stdout --- 22:53:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11562m 8915Mi am-b59c765b7-bgsxp 10817m 8769Mi am-b59c765b7-ch5q4 11308m 8532Mi ds-cts-0 12362m 7852Mi ds-cts-1 11498m 7663Mi ds-cts-2 12867m 7948Mi ds-idrepo-0 5398m 3210Mi ds-idrepo-1 5252m 2839Mi ds-idrepo-2 5296m 2938Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 967Mi idm-b85f9d978-wnhxb 4m 1010Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6737m 2049Mi 22:53:32 DEBUG --- stderr --- 22:53:32 DEBUG 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390813" 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG --- http response --- 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:33 DEBUG { 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG { 22:53:33 DEBUG "data": { 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG "data": { 22:53:33 DEBUG "result": [ 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG { 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "metric": { 22:53:33 DEBUG { 22:53:33 DEBUG { 22:53:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG "metric": { 22:53:33 DEBUG }, 22:53:33 DEBUG "data": { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "instance": "10.142.0.107:9100" 22:53:33 DEBUG 22:53:33 DEBUG { 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "value": [ 22:53:33 DEBUG [...] 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG "instance": "10.142.0.107:9100" 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:33 DEBUG [...] 22:53:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:53:33 DEBUG 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 DEBUG { 22:53:33 DEBUG 22:53:33 DEBUG "value": [ 22:53:33 DEBUG [...] 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG 22:53:33 DEBUG "data": { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "1817.1" 22:53:33 DEBUG 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "instance": "10.142.0.43:9100" 22:53:33 DEBUG ] 22:53:33 DEBUG 22:53:33 DEBUG { 22:53:33 DEBUG }, 22:53:33 DEBUG } 22:53:33 DEBUG "instance": "10.142.0.43:9100" 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "value": [ 22:53:33 DEBUG ] 22:53:33 DEBUG }, 22:53:33 DEBUG "pod": "ds-idrepo-0" 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG } 22:53:33 DEBUG "value": [ 22:53:33 DEBUG }, 22:53:33 DEBUG "60716869.266666666" 22:53:33 DEBUG } 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG ] 22:53:33 DEBUG "0.14750000000000038" 22:53:33 DEBUG 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG [...] 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG 22:53:33 DEBUG } 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:53:33 DEBUG }, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG "0.0000386236188074441" 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390813" 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG { 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG "data": { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "result": [ 22:53:33 DEBUG { 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:53:33 DEBUG }, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:53:33 DEBUG }, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG "2243158016" 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 INFO 22:53:33 INFO [loop_until]: kubectl --namespace=xlou top node 22:53:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 DEBUG --- status code --- 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390813" 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 INFO [loop_until]: OK (rc = 0) 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG --- status code --- 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG --- status code --- 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG --- status code --- 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG --- stdout --- 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG --- http response --- 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG --- status code --- 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5444Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5266m 17% 6134Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11762m 39% 11117Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 2990Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12202m 40% 10104Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 167m 0% 4975Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11956m 40% 11521Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27753m 92% 2923Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11792m 39% 10608Mi 9% gke-xlou-cdm-ds-0f601751-j488 5549m 18% 5513Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11544m 38% 11204Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5854m 19% 5947Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12771m 42% 9627Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6501m 40% 5842Mi 9% 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG --- stderr --- 22:53:33 DEBUG --- http response --- 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 INFO [http_cmd]: http status code OK 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:33 DEBUG --- status code --- 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG { 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG { 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:33 DEBUG { 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG { 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG { 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:53:33 DEBUG { 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG { 22:53:33 DEBUG --- http response --- 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG "data": { 22:53:33 DEBUG { 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG { 22:53:33 DEBUG { 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG "data": { 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG { 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:53:33 DEBUG "data": { 22:53:33 DEBUG { 22:53:33 DEBUG { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG { 22:53:33 DEBUG "data": { 22:53:33 DEBUG { 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG "data": { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG { 22:53:33 DEBUG "data": { 22:53:33 DEBUG { 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG { 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG "data": { 22:53:33 DEBUG ----- output ----- 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "data": { 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG "data": { 22:53:33 DEBUG "data": { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG "data": { 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG { 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "data": { 22:53:33 DEBUG "data": { 22:53:33 DEBUG { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "data": { 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "data": { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "result": [ 22:53:33 DEBUG { 22:53:33 DEBUG "data": { 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "data": { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "data": { 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "status": "success", 22:53:33 DEBUG { 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "result": [ 22:53:33 DEBUG { 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "data": { 22:53:33 DEBUG { 22:53:33 DEBUG "data": { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "instance": "10.142.0.107:9100" 22:53:33 DEBUG { 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "result": [ 22:53:33 DEBUG { 22:53:33 DEBUG { 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "pod": "ds-idrepo-0" 22:53:33 DEBUG }, 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "result": [ 22:53:33 DEBUG { 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "resultType": "vector", 22:53:33 DEBUG "result": [ 22:53:33 DEBUG }, 22:53:33 DEBUG { 22:53:33 DEBUG }, 22:53:33 DEBUG "metric": { 22:53:33 DEBUG { 22:53:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:53:33 DEBUG { 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "pod": "ds-cts-0" 22:53:33 DEBUG { 22:53:33 DEBUG "metric": { 22:53:33 DEBUG { 22:53:33 DEBUG "result": [ 22:53:33 DEBUG "pod": "ds-idrepo-0" 22:53:33 DEBUG "result": [ 22:53:33 DEBUG { 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:53:33 DEBUG "metric": { 22:53:33 DEBUG }, 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "pod": "ds-cts-0" 22:53:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:53:33 DEBUG }, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 22:53:33 DEBUG }, 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "pod": "ds-idrepo-0" 22:53:33 DEBUG "metric": { 22:53:33 DEBUG { 22:53:33 DEBUG }, 22:53:33 DEBUG { 22:53:33 DEBUG "metric": { 22:53:33 DEBUG 22:53:33 DEBUG "pod": "ds-idrepo-0" 22:53:33 DEBUG 22:53:33 DEBUG }, 22:53:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "instance": "10.142.0.107:9100" 22:53:33 DEBUG }, 22:53:33 DEBUG }, 22:53:33 DEBUG { 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "pod": "ds-idrepo-0" 22:53:33 DEBUG }, 22:53:33 DEBUG "instance": "10.142.0.107:9100" 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "metric": { 22:53:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:53:33 DEBUG 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 DEBUG "value": [ 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 DEBUG }, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "metric": { 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG }, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG }, 22:53:33 DEBUG "pod": "ds-idrepo-0" 22:53:33 DEBUG 22:53:33 DEBUG "pod": "ds-cts-0" 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "pod": "ds-idrepo-0" 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 DEBUG }, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 22:53:33 DEBUG }, 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG "pod": "ds-idrepo-2" 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:53:33 DEBUG 22:53:33 DEBUG }, 22:53:33 DEBUG "pod": "ds-cts-2" 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG [...] 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 DEBUG "pod": "ds-cts-2" 22:53:33 DEBUG 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 DEBUG "value": [ 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "pod": "ds-idrepo-2" 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "instance": "10.142.0.43:9100" 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "value": [ 22:53:33 DEBUG [...] 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 DEBUG "instance": "10.142.0.43:9100" 22:53:33 DEBUG }, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 22:53:33 DEBUG "pod": "ds-cts-2" 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG "pod": "ds-idrepo-2" 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG "1816.8666666666666" 22:53:33 DEBUG ] 22:53:33 DEBUG }, 22:53:33 DEBUG 22:53:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:53:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:53:33 DEBUG }, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG "180498082816" 22:53:33 DEBUG ] 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "instance": "10.142.0.43:9100" 22:53:33 DEBUG }, 22:53:33 DEBUG }, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:53:33 DEBUG }, 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG }, 22:53:33 DEBUG } 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG "0" 22:53:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG }, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG 22:53:33 DEBUG 22:53:33 DEBUG } 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG }, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG "0" 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG 22:53:33 DEBUG "0" 22:53:33 DEBUG "value": [ 22:53:33 DEBUG ] 22:53:33 DEBUG "6.6071258985417725" 22:53:33 DEBUG ] 22:53:33 DEBUG }, 22:53:33 DEBUG "0.000033333333333333335" 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG "0" 22:53:33 DEBUG "pod": "ds-cts-2" 22:53:33 DEBUG 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG "60715580.8" 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG "1816.9666666666667" 22:53:33 DEBUG "0.016" 22:53:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:53:33 DEBUG ] 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG "value": [ 22:53:33 DEBUG ] 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG "0" 22:53:33 DEBUG ] 22:53:33 DEBUG }, 22:53:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:53:33 DEBUG "0" 22:53:33 DEBUG ] 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG ] 22:53:33 DEBUG ] 22:53:33 DEBUG ] 22:53:33 DEBUG }, 22:53:33 DEBUG } 22:53:33 DEBUG "0" 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG "value": [ 22:53:33 DEBUG }, 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG "328639.5" 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG "value": [ 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG ] 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG "1829.6" 22:53:33 DEBUG "0.06933333333333333" 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG "value": [ 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG ] 22:53:33 DEBUG ] 22:53:33 DEBUG ] 22:53:33 DEBUG ] 22:53:33 DEBUG ] 22:53:33 DEBUG "0" 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG "0" 22:53:33 DEBUG 1713390813, 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG "0" 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG ] 22:53:33 DEBUG } 22:53:33 DEBUG } 22:53:33 DEBUG } 22:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390872" 22:54:32 ERROR [http_cmd]: ERROR 22:54:32 ERROR --- status code --- 22:54:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:54:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390872" 22:54:32 ERROR --- http response --- 22:54:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:54:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:54:32 INFO [http_cmd]: http status code OK 22:54:32 DEBUG --- status code --- 22:54:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:32 DEBUG --- http response --- 22:54:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:54:33 INFO 22:54:33 INFO [loop_until]: kubectl --namespace=xlou top pods 22:54:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 INFO [loop_until]: OK (rc = 0) 22:54:33 DEBUG --- stdout --- 22:54:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11771m 8919Mi am-b59c765b7-bgsxp 10994m 8768Mi am-b59c765b7-ch5q4 11972m 8534Mi ds-cts-0 12016m 7878Mi ds-cts-1 11816m 7685Mi ds-cts-2 12698m 7928Mi ds-idrepo-0 5282m 3210Mi ds-idrepo-1 5419m 2838Mi ds-idrepo-2 5117m 2938Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 3m 1009Mi lodemon-7cfb6bd874-prp2f 3m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6451m 2051Mi 22:54:33 DEBUG --- stderr --- 22:54:33 DEBUG 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG { 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "data": { 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "result": [ 22:54:33 DEBUG { 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "pod": "ds-idrepo-1" 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "0" 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390873" 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390873" 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG { 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "data": { 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "result": [ 22:54:33 DEBUG { 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:54:33 DEBUG }, 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG "value": [ 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG "1826.1333333333332" 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG ] 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:33 DEBUG } 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:33 DEBUG ] 22:54:33 DEBUG { 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG } 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG { 22:54:33 DEBUG } 22:54:33 DEBUG "data": { 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "data": { 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "result": [ 22:54:33 DEBUG { 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "instance": "10.142.0.107:9100" 22:54:33 DEBUG "result": [ 22:54:33 DEBUG }, 22:54:33 DEBUG { 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "metric": { 22:54:33 DEBUG 22:54:33 DEBUG "instance": "10.142.0.107:9100" 22:54:33 DEBUG 22:54:33 DEBUG }, 22:54:33 DEBUG 22:54:33 DEBUG "value": [ 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG "instance": "10.142.0.43:9100" 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "60778707.233333334" 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG "instance": "10.142.0.43:9100" 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "0.08383333333333383" 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390873" 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG { 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "data": { 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "result": [ 22:54:33 DEBUG { 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "2246750208" 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 INFO 22:54:33 INFO [loop_until]: kubectl --namespace=xlou top node 22:54:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390873" 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG --- http response --- 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG { 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG --- http response --- 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390873" 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG --- http response --- 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:33 DEBUG "data": { 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:33 DEBUG --- status code --- 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 DEBUG { 22:54:33 DEBUG "result": [ 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 DEBUG { 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "data": { 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 DEBUG { 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390873" 22:54:33 DEBUG --- status code --- 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 DEBUG --- http response --- 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 DEBUG "data": { 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 DEBUG "resultType": "vector", 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390873" 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG "metric": { 22:54:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390873" 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 INFO [loop_until]: OK (rc = 0) 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:33 DEBUG "resultType": "vector", 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG "result": [ 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG --- status code --- 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG --- status code --- 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG --- stdout --- 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG "result": [ 22:54:33 DEBUG { 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG { 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG { 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG "metric": { 22:54:33 DEBUG --- status code --- 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG "metric": { 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG }, 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5440Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5237m 17% 6139Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11447m 38% 11112Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3007Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12163m 40% 10100Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 4988Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11967m 40% 11525Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27818m 93% 2922Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11862m 39% 10611Mi 9% gke-xlou-cdm-ds-0f601751-j488 5596m 18% 5515Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11645m 39% 11221Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5892m 19% 5957Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12366m 41% 9618Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6711m 42% 5844Mi 9% 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG --- http response --- 22:54:33 INFO [http_cmd]: http status code OK 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG "pod": "ds-idrepo-1" 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG --- status code --- 22:54:33 DEBUG "pod": "ds-idrepo-0" 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG "value": [ 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:33 DEBUG --- stderr --- 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:33 DEBUG { 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG "data": { 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:33 DEBUG }, 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:33 DEBUG 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:33 DEBUG { 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:54:33 DEBUG --- http response --- 22:54:33 DEBUG "value": [ 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG { 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG { 22:54:33 DEBUG { 22:54:33 DEBUG { 22:54:33 DEBUG "data": { 22:54:33 DEBUG { 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:33 DEBUG "result": [ 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:54:33 DEBUG 22:54:33 DEBUG { 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG { 22:54:33 DEBUG 22:54:33 DEBUG { 22:54:33 DEBUG "data": { 22:54:33 DEBUG { 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG { 22:54:33 DEBUG { 22:54:33 DEBUG 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG { 22:54:33 DEBUG ----- output ----- 22:54:33 DEBUG 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "data": { 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "data": { 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "data": { 22:54:33 DEBUG "data": { 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "result": [ 22:54:33 DEBUG "data": { 22:54:33 DEBUG { 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG [...] 22:54:33 DEBUG { 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG { 22:54:33 DEBUG 22:54:33 DEBUG "data": { 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG [...] 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "data": { 22:54:33 DEBUG "result": [ 22:54:33 DEBUG "data": { 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "result": [ 22:54:33 DEBUG "data": { 22:54:33 DEBUG { 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG "instance": "10.142.0.107:9100" 22:54:33 DEBUG "data": { 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG "data": { 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG [...] 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "result": [ 22:54:33 DEBUG 22:54:33 DEBUG "result": [ 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG { 22:54:33 DEBUG "result": [ 22:54:33 DEBUG { 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "result": [ 22:54:33 DEBUG "data": { 22:54:33 DEBUG }, 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "status": "success", 22:54:33 DEBUG 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "data": { 22:54:33 DEBUG 22:54:33 DEBUG "result": [ 22:54:33 DEBUG { 22:54:33 DEBUG 22:54:33 DEBUG { 22:54:33 DEBUG "result": [ 22:54:33 DEBUG "metric": { 22:54:33 DEBUG { 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "result": [ 22:54:33 DEBUG "result": [ 22:54:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:54:33 DEBUG { 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "result": [ 22:54:33 DEBUG "data": { 22:54:33 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:54:33 DEBUG "result": [ 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG 22:54:33 DEBUG { 22:54:33 DEBUG "metric": { 22:54:33 DEBUG 22:54:33 DEBUG "metric": { 22:54:33 DEBUG { 22:54:33 DEBUG "instance": "10.142.0.107:9100" 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:54:33 DEBUG { 22:54:33 DEBUG { 22:54:33 DEBUG }, 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "result": [ 22:54:33 DEBUG 22:54:33 DEBUG { 22:54:33 DEBUG "resultType": "vector", 22:54:33 DEBUG }, 22:54:33 DEBUG { 22:54:33 DEBUG "result": [ 22:54:33 DEBUG 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "pod": "ds-cts-0" 22:54:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:54:33 DEBUG "pod": "ds-idrepo-1" 22:54:33 DEBUG "metric": { 22:54:33 DEBUG }, 22:54:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:54:33 DEBUG }, 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "pod": "ds-idrepo-0" 22:54:33 DEBUG { 22:54:33 DEBUG 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "result": [ 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "metric": { 22:54:33 DEBUG { 22:54:33 DEBUG "pod": "ds-idrepo-2" 22:54:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:54:33 DEBUG }, 22:54:33 DEBUG }, 22:54:33 DEBUG }, 22:54:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:54:33 DEBUG "value": [ 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "pod": "ds-idrepo-0" 22:54:33 DEBUG "pod": "ds-idrepo-0" 22:54:33 DEBUG 22:54:33 DEBUG }, 22:54:33 DEBUG "metric": { 22:54:33 DEBUG 22:54:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:54:33 DEBUG { 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "pod": "ds-cts-0" 22:54:33 DEBUG "metric": { 22:54:33 DEBUG }, 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "value": [ 22:54:33 DEBUG }, 22:54:33 DEBUG 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 22:54:33 DEBUG }, 22:54:33 DEBUG }, 22:54:33 DEBUG 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "instance": "10.142.0.107:9100" 22:54:33 DEBUG [...] 22:54:33 DEBUG }, 22:54:33 DEBUG "metric": { 22:54:33 DEBUG "0" 22:54:33 DEBUG }, 22:54:33 DEBUG "pod": "ds-cts-0" 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG }, 22:54:33 DEBUG 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "pod": "ds-idrepo-1" 22:54:33 DEBUG ] 22:54:33 DEBUG "value": [ 22:54:33 DEBUG }, 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG "1826.1" 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "0" 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG ] 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG } 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG ] 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG } 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG "instance": "10.142.0.43:9100" 22:54:33 DEBUG ] 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG } 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG ] 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG }, 22:54:33 DEBUG } 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG ] 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG } 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG "value": [ 22:54:33 DEBUG } 22:54:33 DEBUG [...] 22:54:33 DEBUG 22:54:33 DEBUG [...] 22:54:33 DEBUG } 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG } 22:54:33 DEBUG 22:54:33 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:54:33 DEBUG "instance": "10.142.0.43:9100" 22:54:33 DEBUG 22:54:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:54:33 DEBUG 22:54:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:54:33 DEBUG 22:54:33 DEBUG "pod": "ds-idrepo-2" 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG } 22:54:33 DEBUG 22:54:33 DEBUG "pod": "ds-cts-2" 22:54:33 DEBUG 22:54:33 DEBUG }, 22:54:33 DEBUG }, 22:54:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:54:33 DEBUG }, 22:54:33 DEBUG 22:54:33 DEBUG }, 22:54:33 DEBUG 22:54:33 DEBUG }, 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG "60776643.43333334" 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG 22:54:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:54:33 DEBUG }, 22:54:33 DEBUG 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "value": [ 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "pod": "ds-idrepo-2" 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "pod": "ds-cts-2" 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG 22:54:33 DEBUG ] 22:54:33 DEBUG 22:54:33 DEBUG "pod": "ds-cts-2" 22:54:33 DEBUG 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG }, 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG }, 22:54:33 DEBUG "instance": "10.142.0.43:9100" 22:54:33 DEBUG "0" 22:54:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:54:33 DEBUG } 22:54:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:54:33 DEBUG }, 22:54:33 DEBUG "pod": "ds-cts-2" 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "0.00002210444444444444" 22:54:33 DEBUG }, 22:54:33 DEBUG "183395982336" 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "1829.0666666666666" 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "1826.3999999999999" 22:54:33 DEBUG "value": [ 22:54:33 DEBUG }, 22:54:33 DEBUG ] 22:54:33 DEBUG }, 22:54:33 DEBUG ] 22:54:33 DEBUG }, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG }, 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "0" 22:54:33 DEBUG ] 22:54:33 DEBUG "value": [ 22:54:33 DEBUG ] 22:54:33 DEBUG "328478.5" 22:54:33 DEBUG ] 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG ] 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG } 22:54:33 DEBUG "value": [ 22:54:33 DEBUG } 22:54:33 DEBUG "value": [ 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "value": [ 22:54:33 DEBUG "6.592591978922755" 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG "0" 22:54:33 DEBUG } 22:54:33 DEBUG "0" 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG ] 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG } 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "0" 22:54:33 DEBUG 1713390873, 22:54:33 DEBUG "0" 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG "0" 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG ] 22:54:33 DEBUG "0.05266666666666656" 22:54:33 DEBUG } 22:54:33 DEBUG "0.016" 22:54:33 DEBUG "0" 22:54:33 DEBUG ] 22:54:33 DEBUG ] 22:54:33 DEBUG ] 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG ] 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG ] 22:54:33 DEBUG ] 22:54:33 DEBUG ] 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:54:33 DEBUG } 22:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390932" 22:55:32 ERROR [http_cmd]: ERROR 22:55:32 ERROR --- status code --- 22:55:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:55:32 ERROR --- http response --- 22:55:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:55:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:55:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390932" 22:55:32 INFO [http_cmd]: http status code OK 22:55:32 DEBUG --- status code --- 22:55:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:32 DEBUG --- http response --- 22:55:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:55:33 INFO 22:55:33 INFO [loop_until]: kubectl --namespace=xlou top pods 22:55:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG { 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "data": { 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "result": [ 22:55:33 DEBUG { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "pod": "ds-idrepo-0" 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG "0" 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390933" 22:55:33 INFO [loop_until]: OK (rc = 0) 22:55:33 DEBUG --- stdout --- 22:55:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12143m 8919Mi am-b59c765b7-bgsxp 10870m 8768Mi am-b59c765b7-ch5q4 11945m 8535Mi ds-cts-0 12668m 7846Mi ds-cts-1 11377m 7661Mi ds-cts-2 13768m 7933Mi ds-idrepo-0 5362m 3210Mi ds-idrepo-1 5329m 2838Mi ds-idrepo-2 5133m 2939Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 967Mi idm-b85f9d978-wnhxb 4m 1010Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6709m 2053Mi 22:55:33 DEBUG --- stderr --- 22:55:33 DEBUG 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG --- status code --- 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:33 DEBUG { 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG { 22:55:33 DEBUG "data": { 22:55:33 DEBUG { 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "data": { 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "data": { 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG { 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "result": [ 22:55:33 DEBUG { 22:55:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:55:33 DEBUG { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG }, 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "instance": "10.142.0.107:9100" 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "instance": "10.142.0.107:9100" 22:55:33 DEBUG }, 22:55:33 DEBUG 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "instance": "10.142.0.43:9100" 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG }, 22:55:33 DEBUG 22:55:33 DEBUG "1822.6333333333332" 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "instance": "10.142.0.43:9100" 22:55:33 DEBUG ] 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG }, 22:55:33 DEBUG } 22:55:33 DEBUG "0.14996666666666555" 22:55:33 DEBUG "value": [ 22:55:33 DEBUG ] 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG } 22:55:33 DEBUG "60791160.06666666" 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390933" 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG { 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "data": { 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "result": [ 22:55:33 DEBUG { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG "2247864320" 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 INFO 22:55:33 INFO [loop_until]: kubectl --namespace=xlou top node 22:55:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390933" 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 DEBUG --- status code --- 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390933" 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390933" 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG --- status code --- 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG --- http response --- 22:55:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390933" 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG --- http response --- 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 INFO [loop_until]: OK (rc = 0) 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG --- status code --- 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG --- status code --- 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG --- stdout --- 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG { 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 5439Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5279m 17% 6129Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11861m 39% 11112Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 2998Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12563m 42% 10103Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 4978Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11920m 39% 11527Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27613m 92% 2913Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11328m 37% 10627Mi 9% gke-xlou-cdm-ds-0f601751-j488 5518m 18% 5512Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11457m 38% 11215Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5868m 19% 5954Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12904m 43% 9634Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6744m 42% 5843Mi 9% 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG --- http response --- 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG --- http response --- 22:55:33 INFO [http_cmd]: http status code OK 22:55:33 DEBUG { 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG --- stderr --- 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG "data": { 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:33 DEBUG --- status code --- 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:55:33 DEBUG { 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:55:33 DEBUG { 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:33 DEBUG "data": { 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG { 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG "result": [ 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG { 22:55:33 DEBUG --- http response --- 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG "data": { 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG { 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:33 DEBUG { 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:55:33 DEBUG "result": [ 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "data": { 22:55:33 DEBUG "data": { 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG { 22:55:33 DEBUG "data": { 22:55:33 DEBUG "data": { 22:55:33 DEBUG "data": { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG { 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG ----- output ----- 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG "data": { 22:55:33 DEBUG "data": { 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "data": { 22:55:33 DEBUG "data": { 22:55:33 DEBUG "data": { 22:55:33 DEBUG "data": { 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "pod": "ds-idrepo-0" 22:55:33 DEBUG "data": { 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "data": { 22:55:33 DEBUG { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG { 22:55:33 DEBUG "data": { 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "result": [ 22:55:33 DEBUG }, 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "data": { 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "status": "success", 22:55:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:55:33 DEBUG "data": { 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "result": [ 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "data": { 22:55:33 DEBUG }, 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG { 22:55:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG 22:55:33 DEBUG { 22:55:33 DEBUG "result": [ 22:55:33 DEBUG { 22:55:33 DEBUG "resultType": "vector", 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "result": [ 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "pod": "ds-idrepo-0" 22:55:33 DEBUG "pod": "ds-cts-0" 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG }, 22:55:33 DEBUG { 22:55:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:55:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:55:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:55:33 DEBUG 22:55:33 DEBUG "metric": { 22:55:33 DEBUG { 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "result": [ 22:55:33 DEBUG 22:55:33 DEBUG { 22:55:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:55:33 DEBUG "instance": "10.142.0.107:9100" 22:55:33 DEBUG }, 22:55:33 DEBUG }, 22:55:33 DEBUG "pod": "ds-idrepo-0" 22:55:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:55:33 DEBUG "pod": "ds-idrepo-0" 22:55:33 DEBUG "pod": "ds-idrepo-0" 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "metric": { 22:55:33 DEBUG }, 22:55:33 DEBUG }, 22:55:33 DEBUG }, 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "pod": "ds-idrepo-0" 22:55:33 DEBUG { 22:55:33 DEBUG 22:55:33 DEBUG "metric": { 22:55:33 DEBUG "instance": "10.142.0.107:9100" 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "value": [ 22:55:33 DEBUG }, 22:55:33 DEBUG }, 22:55:33 DEBUG }, 22:55:33 DEBUG }, 22:55:33 DEBUG 22:55:33 DEBUG "pod": "ds-cts-0" 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "instance": "10.142.0.107:9100" 22:55:33 DEBUG 22:55:33 DEBUG "pod": "ds-idrepo-0" 22:55:33 DEBUG }, 22:55:33 DEBUG "metric": { 22:55:33 DEBUG 22:55:33 DEBUG }, 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG }, 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG }, 22:55:33 DEBUG 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "pod": "ds-cts-0" 22:55:33 DEBUG [...] 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG }, 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG "pod": "ds-idrepo-1" 22:55:33 DEBUG }, 22:55:33 DEBUG 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG [...] 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG [...] 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG "value": [ 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:55:33 DEBUG [...] 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG [...] 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG }, 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG 22:55:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG "4232.61117445838" 22:55:33 DEBUG 22:55:33 DEBUG [...] 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG "instance": "10.142.0.43:9100" 22:55:33 DEBUG 22:55:33 DEBUG "pod": "ds-cts-2" 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG "pod": "ds-idrepo-1" 22:55:33 DEBUG }, 22:55:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:55:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:55:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG ] 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG }, 22:55:33 DEBUG "pod": "ds-idrepo-2" 22:55:33 DEBUG }, 22:55:33 DEBUG "pod": "ds-idrepo-2" 22:55:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:55:33 DEBUG "pod": "ds-cts-2" 22:55:33 DEBUG 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG }, 22:55:33 DEBUG }, 22:55:33 DEBUG }, 22:55:33 DEBUG 22:55:33 DEBUG 22:55:33 DEBUG "pod": "ds-idrepo-2" 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:55:33 DEBUG "instance": "10.142.0.43:9100" 22:55:33 DEBUG "value": [ 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG }, 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG "pod": "ds-cts-2" 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "instance": "10.142.0.43:9100" 22:55:33 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:55:33 DEBUG }, 22:55:33 DEBUG } 22:55:33 DEBUG 22:55:33 DEBUG "1822.5666666666666" 22:55:33 DEBUG }, 22:55:33 DEBUG }, 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG }, 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG "1820.5333333333333" 22:55:33 DEBUG "1822.4333333333334" 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG }, 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 22:55:33 DEBUG ] 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "186500563968" 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG "0.000055437777777777775" 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG }, 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "328122.5" 22:55:33 DEBUG "0.016" 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "value": [ 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG "pod": "ds-cts-2" 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG "0" 22:55:33 DEBUG ] 22:55:33 DEBUG "0" 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG ] 22:55:33 DEBUG ] 22:55:33 DEBUG "0" 22:55:33 DEBUG ] 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG ] 22:55:33 DEBUG ] 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG "0" 22:55:33 DEBUG }, 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG } 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG "0" 22:55:33 DEBUG "value": [ 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG "60790267.33333333" 22:55:33 DEBUG "0" 22:55:33 DEBUG ] 22:55:33 DEBUG "value": [ 22:55:33 DEBUG "0.07233333333333346" 22:55:33 DEBUG "6.573504999969628" 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG ] 22:55:33 DEBUG "0" 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG ] 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG 1713390933, 22:55:33 DEBUG ] 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG "0" 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG "0" 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG ] 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG ] 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:55:33 DEBUG } 22:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713390992" 22:56:32 ERROR [http_cmd]: ERROR 22:56:32 ERROR --- status code --- 22:56:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:56:32 ERROR --- http response --- 22:56:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:56:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390992" 22:56:32 INFO [http_cmd]: http status code OK 22:56:32 DEBUG --- status code --- 22:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:32 DEBUG --- http response --- 22:56:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:33 DEBUG --- http response --- 22:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:56:33 DEBUG ----- output ----- 22:56:33 DEBUG { 22:56:33 DEBUG "status": "success", 22:56:33 DEBUG "data": { 22:56:33 DEBUG "resultType": "vector", 22:56:33 DEBUG "result": [ 22:56:33 DEBUG { 22:56:33 DEBUG "metric": { 22:56:33 DEBUG "pod": "ds-idrepo-1" 22:56:33 DEBUG }, 22:56:33 DEBUG "value": [ 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG [...] 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:56:33 DEBUG }, 22:56:33 DEBUG "value": [ 22:56:33 DEBUG 1713390993, 22:56:33 DEBUG "0" 22:56:33 DEBUG ] 22:56:33 DEBUG } 22:56:33 DEBUG ] 22:56:33 DEBUG } 22:56:33 DEBUG } 22:56:33 INFO 22:56:33 INFO [loop_until]: kubectl --namespace=xlou top pods 22:56:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390993" 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:33 DEBUG --- http response --- 22:56:33 DEBUG --- http response --- 22:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:33 DEBUG ----- output ----- 22:56:33 DEBUG ----- output ----- 22:56:33 DEBUG { 22:56:33 DEBUG { 22:56:33 DEBUG "status": "success", 22:56:33 DEBUG "status": "success", 22:56:33 DEBUG "data": { 22:56:33 DEBUG "data": { 22:56:33 DEBUG "resultType": "vector", 22:56:33 DEBUG "resultType": "vector", 22:56:33 DEBUG "result": [ 22:56:33 DEBUG "result": [ 22:56:33 DEBUG { 22:56:33 DEBUG { 22:56:33 DEBUG "metric": { 22:56:33 DEBUG "metric": { 22:56:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:56:33 DEBUG "instance": "10.142.0.107:9100" 22:56:33 DEBUG }, 22:56:33 DEBUG }, 22:56:33 DEBUG "value": [ 22:56:33 DEBUG "value": [ 22:56:33 DEBUG 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG --- http response --- 22:56:33 DEBUG 22:56:33 DEBUG [...] 22:56:33 DEBUG 22:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:33 DEBUG 22:56:33 DEBUG [...] 22:56:33 DEBUG ----- output ----- 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG { 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG "status": "success", 22:56:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:56:33 DEBUG 22:56:33 DEBUG "data": { 22:56:33 DEBUG }, 22:56:33 DEBUG "instance": "10.142.0.43:9100" 22:56:33 DEBUG "resultType": "vector", 22:56:33 DEBUG "value": [ 22:56:33 DEBUG }, 22:56:33 DEBUG "result": [ 22:56:33 DEBUG 1713390993, 22:56:33 DEBUG "value": [ 22:56:33 DEBUG { 22:56:33 DEBUG "1821.3666666666666" 22:56:33 DEBUG 1713390993, 22:56:33 DEBUG "metric": { 22:56:33 DEBUG ] 22:56:33 DEBUG "60740091.2" 22:56:33 DEBUG "instance": "10.142.0.107:9100" 22:56:33 DEBUG } 22:56:33 DEBUG ] 22:56:33 DEBUG }, 22:56:33 DEBUG ] 22:56:33 DEBUG } 22:56:33 DEBUG "value": [ 22:56:33 DEBUG } 22:56:33 DEBUG ] 22:56:33 DEBUG 22:56:33 DEBUG } 22:56:33 DEBUG } 22:56:33 DEBUG 22:56:33 DEBUG } 22:56:33 DEBUG 22:56:33 DEBUG [...] 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG "instance": "10.142.0.43:9100" 22:56:33 DEBUG }, 22:56:33 DEBUG "value": [ 22:56:33 DEBUG 1713390993, 22:56:33 DEBUG "0.10013333333333396" 22:56:33 DEBUG ] 22:56:33 DEBUG } 22:56:33 DEBUG ] 22:56:33 DEBUG } 22:56:33 DEBUG } 22:56:33 INFO [loop_until]: OK (rc = 0) 22:56:33 DEBUG --- stdout --- 22:56:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12218m 8919Mi am-b59c765b7-bgsxp 10835m 8766Mi am-b59c765b7-ch5q4 11975m 8533Mi ds-cts-0 12610m 7887Mi ds-cts-1 11470m 7669Mi ds-cts-2 12968m 7936Mi ds-idrepo-0 5355m 3220Mi ds-idrepo-1 5381m 2838Mi ds-idrepo-2 5073m 2938Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1010Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6595m 2056Mi 22:56:33 DEBUG --- stderr --- 22:56:33 DEBUG 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713390993" 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:33 DEBUG --- http response --- 22:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:33 DEBUG ----- output ----- 22:56:33 DEBUG { 22:56:33 DEBUG "status": "success", 22:56:33 DEBUG "data": { 22:56:33 DEBUG "resultType": "vector", 22:56:33 DEBUG "result": [ 22:56:33 DEBUG { 22:56:33 DEBUG "metric": { 22:56:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:56:33 DEBUG }, 22:56:33 DEBUG "value": [ 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG [...] 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:56:33 DEBUG }, 22:56:33 DEBUG "value": [ 22:56:33 DEBUG 1713390993, 22:56:33 DEBUG "2251657216" 22:56:33 DEBUG ] 22:56:33 DEBUG } 22:56:33 DEBUG ] 22:56:33 DEBUG } 22:56:33 DEBUG } 22:56:33 INFO 22:56:33 INFO [loop_until]: kubectl --namespace=xlou top node 22:56:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:33 DEBUG --- http response --- 22:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:56:33 DEBUG ----- output ----- 22:56:33 DEBUG { 22:56:33 DEBUG "status": "success", 22:56:33 DEBUG "data": { 22:56:33 DEBUG "resultType": "vector", 22:56:33 DEBUG "result": [ 22:56:33 DEBUG { 22:56:33 DEBUG "metric": { 22:56:33 DEBUG "pod": "ds-cts-0" 22:56:33 DEBUG }, 22:56:33 DEBUG "value": [ 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG [...] 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG 22:56:33 DEBUG "pod": "ds-cts-2" 22:56:33 DEBUG }, 22:56:33 DEBUG "value": [ 22:56:33 DEBUG 1713390993, 22:56:33 DEBUG "0" 22:56:33 DEBUG ] 22:56:33 DEBUG } 22:56:33 DEBUG ] 22:56:33 DEBUG } 22:56:33 DEBUG } 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 DEBUG --- status code --- 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713390993" 22:56:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713390993" 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:33 DEBUG --- status code --- 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 INFO [loop_until]: OK (rc = 0) 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG --- http response --- 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 DEBUG --- status code --- 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG --- stdout --- 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG --- status code --- 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 INFO [http_cmd]: http status code OK 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:33 DEBUG --- http response --- 22:56:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG --- status code --- 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:34 DEBUG --- http response --- 22:56:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5436Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5193m 17% 6134Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11967m 40% 11110Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3005Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12468m 41% 10120Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 163m 0% 4994Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11641m 38% 11537Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28030m 93% 2941Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11691m 39% 10612Mi 9% gke-xlou-cdm-ds-0f601751-j488 5569m 18% 5521Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11641m 38% 11230Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5936m 19% 5949Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12952m 43% 9630Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6723m 42% 5844Mi 9% 22:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:34 DEBUG --- status code --- 22:56:34 INFO [http_cmd]: http status code OK 22:56:34 DEBUG --- status code --- 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG --- status code --- 22:56:34 DEBUG --- status code --- 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG --- http response --- 22:56:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:34 DEBUG --- stderr --- 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:34 DEBUG --- status code --- 22:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:34 DEBUG { 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG { 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG { 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:56:34 DEBUG --- http response --- 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:56:34 DEBUG { 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:34 DEBUG { 22:56:34 DEBUG "data": { 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG { 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG { 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG { 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG { 22:56:34 DEBUG { 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG "data": { 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG { 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG { 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG { 22:56:34 DEBUG { 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG "data": { 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG { 22:56:34 DEBUG ----- output ----- 22:56:34 DEBUG { 22:56:34 DEBUG "data": { 22:56:34 DEBUG { 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG { 22:56:34 DEBUG "data": { 22:56:34 DEBUG "result": [ 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG "data": { 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG "data": { 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG "data": { 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "data": { 22:56:34 DEBUG "data": { 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG { 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG "data": { 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG { 22:56:34 DEBUG "data": { 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "data": { 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "data": { 22:56:34 DEBUG "data": { 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "result": [ 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "result": [ 22:56:34 DEBUG "data": { 22:56:34 DEBUG "data": { 22:56:34 DEBUG "result": [ 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "result": [ 22:56:34 DEBUG "data": { 22:56:34 DEBUG "result": [ 22:56:34 DEBUG "metric": { 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "result": [ 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "result": [ 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "result": [ 22:56:34 DEBUG { 22:56:34 DEBUG "result": [ 22:56:34 DEBUG "status": "success", 22:56:34 DEBUG { 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG { 22:56:34 DEBUG "result": [ 22:56:34 DEBUG { 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG { 22:56:34 DEBUG "pod": "ds-idrepo-1" 22:56:34 DEBUG "result": [ 22:56:34 DEBUG { 22:56:34 DEBUG "result": [ 22:56:34 DEBUG { 22:56:34 DEBUG "result": [ 22:56:34 DEBUG { 22:56:34 DEBUG { 22:56:34 DEBUG "metric": { 22:56:34 DEBUG { 22:56:34 DEBUG "data": { 22:56:34 DEBUG "metric": { 22:56:34 DEBUG "result": [ 22:56:34 DEBUG "result": [ 22:56:34 DEBUG "metric": { 22:56:34 DEBUG { 22:56:34 DEBUG "metric": { 22:56:34 DEBUG "metric": { 22:56:34 DEBUG "metric": { 22:56:34 DEBUG }, 22:56:34 DEBUG { 22:56:34 DEBUG "metric": { 22:56:34 DEBUG { 22:56:34 DEBUG "metric": { 22:56:34 DEBUG "result": [ 22:56:34 DEBUG "metric": { 22:56:34 DEBUG "metric": { 22:56:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:56:34 DEBUG "metric": { 22:56:34 DEBUG "resultType": "vector", 22:56:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:56:34 DEBUG { 22:56:34 DEBUG { 22:56:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:56:34 DEBUG "result": [ 22:56:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:56:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:56:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:56:34 DEBUG "value": [ 22:56:34 DEBUG "metric": { 22:56:34 DEBUG "pod": "ds-idrepo-0" 22:56:34 DEBUG "metric": { 22:56:34 DEBUG "pod": "ds-idrepo-0" 22:56:34 DEBUG { 22:56:34 DEBUG "pod": "ds-idrepo-1" 22:56:34 DEBUG }, 22:56:34 DEBUG }, 22:56:34 DEBUG "instance": "10.142.0.107:9100" 22:56:34 DEBUG "result": [ 22:56:34 DEBUG }, 22:56:34 DEBUG "metric": { 22:56:34 DEBUG "metric": { 22:56:34 DEBUG }, 22:56:34 DEBUG { 22:56:34 DEBUG }, 22:56:34 DEBUG }, 22:56:34 DEBUG }, 22:56:34 DEBUG 22:56:34 DEBUG }, 22:56:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:56:34 DEBUG "pod": "ds-idrepo-1" 22:56:34 DEBUG }, 22:56:34 DEBUG "value": [ 22:56:34 DEBUG "value": [ 22:56:34 DEBUG 22:56:34 DEBUG "value": [ 22:56:34 DEBUG }, 22:56:34 DEBUG { 22:56:34 DEBUG "value": [ 22:56:34 DEBUG "pod": "ds-cts-0" 22:56:34 DEBUG "pod": "ds-idrepo-0" 22:56:34 DEBUG "value": [ 22:56:34 DEBUG "metric": { 22:56:34 DEBUG "value": [ 22:56:34 DEBUG "value": [ 22:56:34 DEBUG "value": [ 22:56:34 DEBUG 22:56:34 DEBUG "value": [ 22:56:34 DEBUG }, 22:56:34 DEBUG }, 22:56:34 DEBUG 22:56:34 DEBUG "metric": { 22:56:34 DEBUG 22:56:34 DEBUG "pod": "ds-cts-0" 22:56:34 DEBUG 22:56:34 DEBUG "value": [ 22:56:34 DEBUG "metric": { 22:56:34 DEBUG 22:56:34 DEBUG }, 22:56:34 DEBUG }, 22:56:34 DEBUG 22:56:34 DEBUG "instance": "10.142.0.107:9100" 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG "value": [ 22:56:34 DEBUG "value": [ 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG }, 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG "instance": "10.142.0.107:9100" 22:56:34 DEBUG 22:56:34 DEBUG "value": [ 22:56:34 DEBUG "value": [ 22:56:34 DEBUG 22:56:34 DEBUG }, 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG [...] 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG "pod": "ds-idrepo-0" 22:56:34 DEBUG [...] 22:56:34 DEBUG 22:56:34 DEBUG "value": [ 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG }, 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG "value": [ 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG }, 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG [...] 22:56:34 DEBUG [...] 22:56:34 DEBUG 22:56:34 DEBUG "value": [ 22:56:34 DEBUG [...] 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG [...] 22:56:34 DEBUG 22:56:34 DEBUG [...] 22:56:34 DEBUG [...] 22:56:34 DEBUG [...] 22:56:34 DEBUG 22:56:34 DEBUG [...] 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG "value": [ 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG [...] 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG [...] 22:56:34 DEBUG [...] 22:56:34 DEBUG 22:56:34 DEBUG "pod": "ds-idrepo-2" 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG [...] 22:56:34 DEBUG [...] 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG }, 22:56:34 DEBUG [...] 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG [...] 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG }, 22:56:34 DEBUG 22:56:34 DEBUG "pod": "ds-idrepo-2" 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG "value": [ 22:56:34 DEBUG 22:56:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:56:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:56:34 DEBUG 22:56:34 DEBUG [...] 22:56:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:56:34 DEBUG 22:56:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:56:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:56:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:56:34 DEBUG "value": [ 22:56:34 DEBUG 22:56:34 DEBUG }, 22:56:34 DEBUG 22:56:34 DEBUG [...] 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG 22:56:34 DEBUG }, 22:56:34 DEBUG }, 22:56:34 DEBUG "instance": "10.142.0.43:9100" 22:56:34 DEBUG 22:56:34 DEBUG }, 22:56:34 DEBUG 22:56:34 DEBUG 22:56:34 DEBUG }, 22:56:34 DEBUG 22:56:34 DEBUG }, 22:56:34 DEBUG "value": [ 22:56:34 DEBUG }, 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG 22:56:34 DEBUG "value": [ 22:56:34 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 22:56:34 DEBUG 22:56:34 DEBUG "0" 22:56:34 DEBUG 22:56:34 DEBUG "value": [ 22:56:34 DEBUG "value": [ 22:56:34 DEBUG }, 22:56:34 DEBUG 22:56:34 DEBUG "value": [ 22:56:34 DEBUG "pod": "ds-cts-2" 22:56:34 DEBUG "pod": "ds-cts-2" 22:56:34 DEBUG "value": [ 22:56:34 DEBUG 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG }, 22:56:34 DEBUG "value": [ 22:56:34 DEBUG "0" 22:56:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG }, 22:56:34 DEBUG 22:56:34 DEBUG ] 22:56:34 DEBUG "pod": "ds-cts-2" 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG "value": [ 22:56:34 DEBUG 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG }, 22:56:34 DEBUG "value": [ 22:56:34 DEBUG }, 22:56:34 DEBUG "instance": "10.142.0.43:9100" 22:56:34 DEBUG "6.57830280402922" 22:56:34 DEBUG "value": [ 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG ] 22:56:34 DEBUG }, 22:56:34 DEBUG "0" 22:56:34 DEBUG "value": [ 22:56:34 DEBUG 22:56:34 DEBUG } 22:56:34 DEBUG }, 22:56:34 DEBUG "0" 22:56:34 DEBUG "0" 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG "instance": "10.142.0.43:9100" 22:56:34 DEBUG "0.017" 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG "value": [ 22:56:34 DEBUG }, 22:56:34 DEBUG ] 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG "1821.1666666666667" 22:56:34 DEBUG ] 22:56:34 DEBUG "value": [ 22:56:34 DEBUG ] 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG "pod": "ds-idrepo-2" 22:56:34 DEBUG ] 22:56:34 DEBUG "value": [ 22:56:34 DEBUG ] 22:56:34 DEBUG ] 22:56:34 DEBUG "189177213952" 22:56:34 DEBUG }, 22:56:34 DEBUG ] 22:56:34 DEBUG "0" 22:56:34 DEBUG "1816.8333333333333" 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG "value": [ 22:56:34 DEBUG } 22:56:34 DEBUG "1821" 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG } 22:56:34 DEBUG "0" 22:56:34 DEBUG }, 22:56:34 DEBUG } 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG "value": [ 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG ] 22:56:34 DEBUG "0" 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG ] 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG "328000" 22:56:34 DEBUG ] 22:56:34 DEBUG ] 22:56:34 DEBUG "value": [ 22:56:34 DEBUG } 22:56:34 DEBUG "0.00002210444444444444" 22:56:34 DEBUG ] 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG "0.06099999999999992" 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG 1713390993, 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG "60739339.83333333" 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG "0" 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG ] 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:56:34 DEBUG } 22:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391052" 22:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391052" 22:57:32 ERROR [http_cmd]: ERROR 22:57:32 ERROR --- status code --- 22:57:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:57:32 ERROR --- http response --- 22:57:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:57:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:57:32 INFO [http_cmd]: http status code OK 22:57:32 DEBUG --- status code --- 22:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:32 DEBUG --- http response --- 22:57:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391053" 22:57:33 INFO [http_cmd]: http status code OK 22:57:33 DEBUG --- status code --- 22:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:33 DEBUG --- http response --- 22:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:57:33 DEBUG ----- output ----- 22:57:33 DEBUG { 22:57:33 DEBUG "status": "success", 22:57:33 DEBUG "data": { 22:57:33 DEBUG "resultType": "vector", 22:57:33 DEBUG "result": [ 22:57:33 DEBUG { 22:57:33 DEBUG "metric": { 22:57:33 DEBUG "pod": "ds-idrepo-0" 22:57:33 DEBUG }, 22:57:33 DEBUG "value": [ 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 DEBUG [...] 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 DEBUG "pod": "am-b59c765b7-7xg4f" 22:57:33 DEBUG }, 22:57:33 DEBUG "value": [ 22:57:33 DEBUG 1713391053, 22:57:33 DEBUG "12.013798305730024" 22:57:33 DEBUG ] 22:57:33 DEBUG } 22:57:33 DEBUG ] 22:57:33 DEBUG } 22:57:33 DEBUG } 22:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391053" 22:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391053" 22:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391053" 22:57:33 INFO [http_cmd]: http status code OK 22:57:33 DEBUG --- status code --- 22:57:33 INFO [http_cmd]: http status code OK 22:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:33 DEBUG --- status code --- 22:57:33 DEBUG --- http response --- 22:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:33 DEBUG --- http response --- 22:57:33 DEBUG ----- output ----- 22:57:33 DEBUG { 22:57:33 DEBUG "status": "success", 22:57:33 DEBUG "data": { 22:57:33 DEBUG "resultType": "vector", 22:57:33 DEBUG "result": [ 22:57:33 DEBUG { 22:57:33 DEBUG "metric": { 22:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:57:33 DEBUG ----- output ----- 22:57:33 DEBUG }, 22:57:33 DEBUG { 22:57:33 DEBUG "value": [ 22:57:33 DEBUG "status": "success", 22:57:33 DEBUG 22:57:33 DEBUG "data": { 22:57:33 DEBUG 22:57:33 DEBUG "resultType": "vector", 22:57:33 DEBUG "result": [ 22:57:33 INFO [http_cmd]: http status code OK 22:57:33 DEBUG { 22:57:33 DEBUG 22:57:33 DEBUG --- status code --- 22:57:33 DEBUG "metric": { 22:57:33 DEBUG [...] 22:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:33 DEBUG "instance": "10.142.0.107:9100" 22:57:33 DEBUG }, 22:57:33 DEBUG "value": [ 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 DEBUG [...] 22:57:33 DEBUG 22:57:33 DEBUG --- http response --- 22:57:33 DEBUG 22:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:33 DEBUG 22:57:33 INFO 22:57:33 DEBUG 22:57:33 DEBUG ----- output ----- 22:57:33 DEBUG 22:57:33 INFO [loop_until]: kubectl --namespace=xlou top pods 22:57:33 DEBUG 22:57:33 DEBUG { 22:57:33 DEBUG "instance": "10.142.0.43:9100" 22:57:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:57:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:57:33 DEBUG "status": "success", 22:57:33 DEBUG }, 22:57:33 DEBUG }, 22:57:33 DEBUG "data": { 22:57:33 DEBUG "value": [ 22:57:33 DEBUG "value": [ 22:57:33 DEBUG "resultType": "vector", 22:57:33 DEBUG 1713391053, 22:57:33 DEBUG 1713391053, 22:57:33 DEBUG "result": [ 22:57:33 DEBUG "60788179.83333333" 22:57:33 DEBUG "1844.0666666666666" 22:57:33 DEBUG { 22:57:33 DEBUG ] 22:57:33 DEBUG ] 22:57:33 DEBUG "metric": { 22:57:33 DEBUG } 22:57:33 DEBUG } 22:57:33 DEBUG "instance": "10.142.0.107:9100" 22:57:33 DEBUG ] 22:57:33 DEBUG ] 22:57:33 DEBUG }, 22:57:33 DEBUG } 22:57:33 DEBUG } 22:57:33 DEBUG "value": [ 22:57:33 DEBUG } 22:57:33 DEBUG } 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391053" 22:57:33 DEBUG [...] 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 DEBUG "instance": "10.142.0.43:9100" 22:57:33 DEBUG }, 22:57:33 DEBUG "value": [ 22:57:33 DEBUG 1713391053, 22:57:33 DEBUG "0.17026666666666868" 22:57:33 DEBUG ] 22:57:33 DEBUG } 22:57:33 DEBUG ] 22:57:33 DEBUG } 22:57:33 DEBUG } 22:57:33 INFO [http_cmd]: http status code OK 22:57:33 DEBUG --- status code --- 22:57:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:33 DEBUG --- http response --- 22:57:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:33 DEBUG ----- output ----- 22:57:33 DEBUG { 22:57:33 DEBUG "status": "success", 22:57:33 DEBUG "data": { 22:57:33 DEBUG "resultType": "vector", 22:57:33 DEBUG "result": [ 22:57:33 DEBUG { 22:57:33 DEBUG "metric": { 22:57:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:57:33 DEBUG }, 22:57:33 DEBUG "value": [ 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 DEBUG [...] 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 DEBUG 22:57:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:57:33 DEBUG }, 22:57:33 DEBUG "value": [ 22:57:33 DEBUG 1713391053, 22:57:33 DEBUG "2250305536" 22:57:33 DEBUG ] 22:57:33 DEBUG } 22:57:33 DEBUG ] 22:57:33 DEBUG } 22:57:33 DEBUG } 22:57:33 INFO [loop_until]: OK (rc = 0) 22:57:33 DEBUG --- stdout --- 22:57:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11802m 8918Mi am-b59c765b7-bgsxp 11097m 8769Mi am-b59c765b7-ch5q4 12076m 8536Mi ds-cts-0 12463m 7872Mi ds-cts-1 11843m 7672Mi ds-cts-2 13152m 7941Mi ds-idrepo-0 5309m 3220Mi ds-idrepo-1 5250m 2838Mi ds-idrepo-2 5071m 2940Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 3m 1009Mi lodemon-7cfb6bd874-prp2f 3m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6551m 2059Mi 22:57:33 DEBUG --- stderr --- 22:57:33 DEBUG 22:57:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391053" 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG { 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "data": { 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "result": [ 22:57:34 DEBUG { 22:57:34 DEBUG "metric": { 22:57:34 DEBUG "pod": "ds-cts-0" 22:57:34 DEBUG }, 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG "pod": "ds-cts-2" 22:57:34 DEBUG }, 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 1713391053, 22:57:34 DEBUG "18.366666666666667" 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO 22:57:34 INFO [loop_until]: kubectl --namespace=xlou top node 22:57:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG { 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "data": { 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "result": [ 22:57:34 DEBUG { 22:57:34 DEBUG "metric": { 22:57:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:57:34 DEBUG }, 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:57:34 DEBUG }, 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG "1826.5" 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391054" 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 DEBUG --- status code --- 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 DEBUG --- status code --- 22:57:34 INFO [loop_until]: OK (rc = 0) 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG --- status code --- 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG --- status code --- 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 DEBUG --- status code --- 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG --- stdout --- 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 INFO [http_cmd]: http status code OK 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5431Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5272m 17% 6144Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11979m 40% 11113Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 2998Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12529m 41% 10111Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 137m 0% 4970Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11915m 39% 11520Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27970m 93% 2947Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11842m 39% 10612Mi 9% gke-xlou-cdm-ds-0f601751-j488 5498m 18% 5522Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11581m 38% 11220Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5862m 19% 5964Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13231m 44% 9638Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6727m 42% 5841Mi 9% 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG --- status code --- 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:34 DEBUG --- stderr --- 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG { 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:34 DEBUG --- http response --- 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:57:34 DEBUG { 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:34 DEBUG { 22:57:34 DEBUG { 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:34 DEBUG { 22:57:34 DEBUG { 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG { 22:57:34 DEBUG { 22:57:34 DEBUG { 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG { 22:57:34 DEBUG { 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "data": { 22:57:34 DEBUG { 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "data": { 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "data": { 22:57:34 DEBUG { 22:57:34 DEBUG ----- output ----- 22:57:34 DEBUG { 22:57:34 DEBUG "data": { 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG { 22:57:34 DEBUG "data": { 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG { 22:57:34 DEBUG "data": { 22:57:34 DEBUG { 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "data": { 22:57:34 DEBUG "data": { 22:57:34 DEBUG "data": { 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG { 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "data": { 22:57:34 DEBUG "data": { 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "data": { 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "data": { 22:57:34 DEBUG "status": "success", 22:57:34 DEBUG "data": { 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "data": { 22:57:34 DEBUG "result": [ 22:57:34 DEBUG { 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "data": { 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "data": { 22:57:34 DEBUG { 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "result": [ 22:57:34 DEBUG { 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "data": { 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG { 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG { 22:57:34 DEBUG "metric": { 22:57:34 DEBUG "pod": "ds-idrepo-0" 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG { 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "metric": { 22:57:34 DEBUG { 22:57:34 DEBUG { 22:57:34 DEBUG { 22:57:34 DEBUG "metric": { 22:57:34 DEBUG "pod": "ds-cts-0" 22:57:34 DEBUG "resultType": "vector", 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "metric": { 22:57:34 DEBUG { 22:57:34 DEBUG { 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "metric": { 22:57:34 DEBUG }, 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "metric": { 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:57:34 DEBUG "metric": { 22:57:34 DEBUG "metric": { 22:57:34 DEBUG }, 22:57:34 DEBUG "metric": { 22:57:34 DEBUG "result": [ 22:57:34 DEBUG { 22:57:34 DEBUG { 22:57:34 DEBUG "instance": "10.142.0.107:9100" 22:57:34 DEBUG "metric": { 22:57:34 DEBUG "metric": { 22:57:34 DEBUG { 22:57:34 DEBUG "metric": { 22:57:34 DEBUG "value": [ 22:57:34 DEBUG { 22:57:34 DEBUG { 22:57:34 DEBUG "pod": "ds-idrepo-0" 22:57:34 DEBUG { 22:57:34 DEBUG }, 22:57:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:57:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "pod": "ds-idrepo-0" 22:57:34 DEBUG "result": [ 22:57:34 DEBUG "metric": { 22:57:34 DEBUG "metric": { 22:57:34 DEBUG "pod": "ds-idrepo-0" 22:57:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:57:34 DEBUG "pod": "ds-idrepo-0" 22:57:34 DEBUG "pod": "ds-idrepo-0" 22:57:34 DEBUG }, 22:57:34 DEBUG 22:57:34 DEBUG "metric": { 22:57:34 DEBUG "metric": { 22:57:34 DEBUG }, 22:57:34 DEBUG "metric": { 22:57:34 DEBUG "value": [ 22:57:34 DEBUG }, 22:57:34 DEBUG }, 22:57:34 DEBUG 22:57:34 DEBUG }, 22:57:34 DEBUG { 22:57:34 DEBUG "instance": "10.142.0.107:9100" 22:57:34 DEBUG }, 22:57:34 DEBUG }, 22:57:34 DEBUG }, 22:57:34 DEBUG }, 22:57:34 DEBUG "pod": "ds-idrepo-0" 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 22:57:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:57:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "pod": "ds-cts-0" 22:57:34 DEBUG 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "metric": { 22:57:34 DEBUG }, 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "value": [ 22:57:34 DEBUG }, 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG }, 22:57:34 DEBUG }, 22:57:34 DEBUG 22:57:34 DEBUG }, 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG "instance": "10.142.0.107:9100" 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG }, 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG [...] 22:57:34 DEBUG [...] 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:57:34 DEBUG [...] 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG "pod": "ds-cts-2" 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG }, 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG }, 22:57:34 DEBUG 22:57:34 DEBUG [...] 22:57:34 DEBUG 22:57:34 DEBUG "pod": "ds-cts-2" 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG "pod": "ds-idrepo-2" 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:57:34 DEBUG 22:57:34 DEBUG }, 22:57:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:57:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "pod": "ds-idrepo-2" 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG }, 22:57:34 DEBUG 22:57:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:57:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:57:34 DEBUG 22:57:34 DEBUG }, 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG 22:57:34 DEBUG 22:57:34 DEBUG }, 22:57:34 DEBUG 22:57:34 DEBUG "value": [ 22:57:34 DEBUG }, 22:57:34 DEBUG }, 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG }, 22:57:34 DEBUG 22:57:34 DEBUG "instance": "10.142.0.43:9100" 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "instance": "10.142.0.43:9100" 22:57:34 DEBUG }, 22:57:34 DEBUG }, 22:57:34 DEBUG "pod": "ds-idrepo-2" 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "0" 22:57:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:57:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "pod": "ds-cts-2" 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "0" 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 22:57:34 DEBUG }, 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG }, 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "value": [ 22:57:34 DEBUG }, 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG ] 22:57:34 DEBUG }, 22:57:34 DEBUG }, 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG }, 22:57:34 DEBUG "6.619872006940272" 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG ] 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG "instance": "10.142.0.43:9100" 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "0" 22:57:34 DEBUG "value": [ 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "0" 22:57:34 DEBUG } 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "0" 22:57:34 DEBUG "value": [ 22:57:34 DEBUG ] 22:57:34 DEBUG "0" 22:57:34 DEBUG "1843.5333333333333" 22:57:34 DEBUG } 22:57:34 DEBUG "0" 22:57:34 DEBUG }, 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG ] 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG "1844" 22:57:34 DEBUG "0" 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG ] 22:57:34 DEBUG ] 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG ] 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG "value": [ 22:57:34 DEBUG "60786614.599999994" 22:57:34 DEBUG } 22:57:34 DEBUG "192578735104" 22:57:34 DEBUG ] 22:57:34 DEBUG ] 22:57:34 DEBUG "0" 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG "328330" 22:57:34 DEBUG } 22:57:34 DEBUG "0.000033333333333333335" 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG 1713391054, 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG "0.018" 22:57:34 DEBUG ] 22:57:34 DEBUG ] 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG "0.08833333333333342" 22:57:34 DEBUG ] 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG ] 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:57:34 DEBUG } 22:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391112" 22:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391112" 22:58:32 ERROR [http_cmd]: ERROR 22:58:32 ERROR --- status code --- 22:58:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:58:32 ERROR --- http response --- 22:58:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:58:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:58:32 INFO [http_cmd]: http status code OK 22:58:32 DEBUG --- status code --- 22:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:32 DEBUG --- http response --- 22:58:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391113" 22:58:33 INFO [http_cmd]: http status code OK 22:58:33 DEBUG --- status code --- 22:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:33 DEBUG --- http response --- 22:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:33 DEBUG ----- output ----- 22:58:33 DEBUG { 22:58:33 DEBUG "status": "success", 22:58:33 DEBUG "data": { 22:58:33 DEBUG "resultType": "vector", 22:58:33 DEBUG "result": [ 22:58:33 DEBUG { 22:58:33 DEBUG "metric": { 22:58:33 DEBUG "pod": "ds-idrepo-0" 22:58:33 DEBUG }, 22:58:33 DEBUG "value": [ 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG [...] 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:58:33 DEBUG }, 22:58:33 DEBUG "value": [ 22:58:33 DEBUG 1713391113, 22:58:33 DEBUG "0.0000312106044756586" 22:58:33 DEBUG ] 22:58:33 DEBUG } 22:58:33 DEBUG ] 22:58:33 DEBUG } 22:58:33 DEBUG } 22:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391113" 22:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391113" 22:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391113" 22:58:33 INFO [http_cmd]: http status code OK 22:58:33 DEBUG --- status code --- 22:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:33 DEBUG --- http response --- 22:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:33 DEBUG ----- output ----- 22:58:33 DEBUG { 22:58:33 DEBUG "status": "success", 22:58:33 DEBUG "data": { 22:58:33 DEBUG "resultType": "vector", 22:58:33 DEBUG "result": [ 22:58:33 DEBUG { 22:58:33 DEBUG "metric": { 22:58:33 DEBUG "instance": "10.142.0.107:9100" 22:58:33 DEBUG }, 22:58:33 DEBUG "value": [ 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG [...] 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG "instance": "10.142.0.43:9100" 22:58:33 DEBUG }, 22:58:33 DEBUG "value": [ 22:58:33 DEBUG 1713391113, 22:58:33 DEBUG "0.08589999999999993" 22:58:33 DEBUG ] 22:58:33 DEBUG } 22:58:33 DEBUG ] 22:58:33 DEBUG } 22:58:33 DEBUG } 22:58:33 INFO [http_cmd]: http status code OK 22:58:33 DEBUG --- status code --- 22:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:33 DEBUG --- http response --- 22:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:33 DEBUG ----- output ----- 22:58:33 DEBUG { 22:58:33 DEBUG "status": "success", 22:58:33 DEBUG "data": { 22:58:33 DEBUG "resultType": "vector", 22:58:33 DEBUG "result": [ 22:58:33 DEBUG { 22:58:33 DEBUG "metric": { 22:58:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:58:33 DEBUG }, 22:58:33 DEBUG "value": [ 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG [...] 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:58:33 DEBUG }, 22:58:33 DEBUG "value": [ 22:58:33 DEBUG 1713391113, 22:58:33 DEBUG "1811.5666666666666" 22:58:33 DEBUG ] 22:58:33 DEBUG } 22:58:33 DEBUG ] 22:58:33 DEBUG } 22:58:33 DEBUG } 22:58:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391113" 22:58:33 INFO [http_cmd]: http status code OK 22:58:33 DEBUG --- status code --- 22:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:33 DEBUG --- http response --- 22:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:33 DEBUG ----- output ----- 22:58:33 DEBUG { 22:58:33 DEBUG "status": "success", 22:58:33 DEBUG "data": { 22:58:33 DEBUG "resultType": "vector", 22:58:33 DEBUG "result": [ 22:58:33 DEBUG { 22:58:33 DEBUG "metric": { 22:58:33 DEBUG "instance": "10.142.0.107:9100" 22:58:33 DEBUG }, 22:58:33 DEBUG "value": [ 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG [...] 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG "instance": "10.142.0.43:9100" 22:58:33 DEBUG }, 22:58:33 DEBUG "value": [ 22:58:33 DEBUG 1713391113, 22:58:33 DEBUG "60638537.3" 22:58:33 DEBUG ] 22:58:33 DEBUG } 22:58:33 DEBUG ] 22:58:33 DEBUG } 22:58:33 DEBUG } 22:58:33 INFO [http_cmd]: http status code OK 22:58:33 DEBUG --- status code --- 22:58:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:33 DEBUG --- http response --- 22:58:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:33 DEBUG ----- output ----- 22:58:33 DEBUG { 22:58:33 DEBUG "status": "success", 22:58:33 DEBUG "data": { 22:58:33 DEBUG "resultType": "vector", 22:58:33 DEBUG "result": [ 22:58:33 DEBUG { 22:58:33 DEBUG "metric": { 22:58:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:58:33 DEBUG }, 22:58:33 DEBUG "value": [ 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG [...] 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG 22:58:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:58:33 DEBUG }, 22:58:33 DEBUG "value": [ 22:58:33 DEBUG 1713391113, 22:58:33 DEBUG "2253508608" 22:58:33 DEBUG ] 22:58:33 DEBUG } 22:58:33 DEBUG ] 22:58:33 DEBUG } 22:58:33 DEBUG } 22:58:33 INFO 22:58:33 INFO [loop_until]: kubectl --namespace=xlou top pods 22:58:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:58:33 INFO [loop_until]: OK (rc = 0) 22:58:33 DEBUG --- stdout --- 22:58:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11852m 8918Mi am-b59c765b7-bgsxp 10865m 8768Mi am-b59c765b7-ch5q4 12175m 8537Mi ds-cts-0 11520m 7884Mi ds-cts-1 11601m 7690Mi ds-cts-2 12890m 7944Mi ds-idrepo-0 5306m 3220Mi ds-idrepo-1 5298m 2855Mi ds-idrepo-2 5057m 2931Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1010Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6573m 2058Mi 22:58:33 DEBUG --- stderr --- 22:58:33 DEBUG 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG { 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG "data": { 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "result": [ 22:58:34 DEBUG { 22:58:34 DEBUG "metric": { 22:58:34 DEBUG "pod": "ds-cts-0" 22:58:34 DEBUG }, 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG "pod": "ds-cts-2" 22:58:34 DEBUG }, 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG "29.933333333333334" 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG { 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG "data": { 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "result": [ 22:58:34 DEBUG { 22:58:34 DEBUG "metric": { 22:58:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:58:34 DEBUG }, 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:58:34 DEBUG }, 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG "1821.8666666666666" 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 INFO 22:58:34 INFO [loop_until]: kubectl --namespace=xlou top node 22:58:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391114" 22:58:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391114" 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG --- http response --- 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG --- status code --- 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG --- status code --- 22:58:34 INFO [loop_until]: OK (rc = 0) 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG --- stdout --- 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 INFO [http_cmd]: http status code OK 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5443Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5207m 17% 6154Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11954m 40% 11121Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3005Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12202m 40% 10112Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 4956Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11898m 39% 11530Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27979m 93% 2926Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11737m 39% 10612Mi 9% gke-xlou-cdm-ds-0f601751-j488 5539m 18% 5513Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11363m 38% 11239Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5868m 19% 5978Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13126m 43% 9647Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6719m 42% 5842Mi 9% 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG --- status code --- 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:34 DEBUG --- stderr --- 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG { 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG { 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:34 DEBUG { 22:58:34 DEBUG { 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:34 DEBUG { 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG --- http response --- 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG { 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:34 DEBUG { 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG { 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:34 DEBUG { 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG "data": { 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG "data": { 22:58:34 DEBUG { 22:58:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:34 DEBUG "data": { 22:58:34 DEBUG "data": { 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG { 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG { 22:58:34 DEBUG { 22:58:34 DEBUG "data": { 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG { 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG { 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG ----- output ----- 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "data": { 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG { 22:58:34 DEBUG "data": { 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "data": { 22:58:34 DEBUG { 22:58:34 DEBUG "data": { 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "data": { 22:58:34 DEBUG "data": { 22:58:34 DEBUG { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "data": { 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "data": { 22:58:34 DEBUG "data": { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "data": { 22:58:34 DEBUG { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "status": "success", 22:58:34 DEBUG { 22:58:34 DEBUG { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "data": { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "data": { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "metric": { 22:58:34 DEBUG { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "data": { 22:58:34 DEBUG "metric": { 22:58:34 DEBUG "metric": { 22:58:34 DEBUG { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "metric": { 22:58:34 DEBUG { 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:58:34 DEBUG "metric": { 22:58:34 DEBUG { 22:58:34 DEBUG { 22:58:34 DEBUG "resultType": "vector", 22:58:34 DEBUG "pod": "ds-idrepo-0" 22:58:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:58:34 DEBUG "metric": { 22:58:34 DEBUG { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "metric": { 22:58:34 DEBUG { 22:58:34 DEBUG { 22:58:34 DEBUG "pod": "ds-idrepo-0" 22:58:34 DEBUG "metric": { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG "metric": { 22:58:34 DEBUG { 22:58:34 DEBUG }, 22:58:34 DEBUG "pod": "ds-cts-0" 22:58:34 DEBUG "metric": { 22:58:34 DEBUG "metric": { 22:58:34 DEBUG "result": [ 22:58:34 DEBUG }, 22:58:34 DEBUG }, 22:58:34 DEBUG "pod": "ds-idrepo-0" 22:58:34 DEBUG "metric": { 22:58:34 DEBUG { 22:58:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:58:34 DEBUG "metric": { 22:58:34 DEBUG "metric": { 22:58:34 DEBUG }, 22:58:34 DEBUG "instance": "10.142.0.107:9100" 22:58:34 DEBUG { 22:58:34 DEBUG "pod": "ds-idrepo-0" 22:58:34 DEBUG "metric": { 22:58:34 DEBUG "value": [ 22:58:34 DEBUG }, 22:58:34 DEBUG "pod": "ds-idrepo-0" 22:58:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:58:34 DEBUG { 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "value": [ 22:58:34 DEBUG }, 22:58:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:58:34 DEBUG "metric": { 22:58:34 DEBUG "instance": "10.142.0.107:9100" 22:58:34 DEBUG "pod": "ds-cts-0" 22:58:34 DEBUG "pod": "ds-idrepo-0" 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 22:58:34 DEBUG "metric": { 22:58:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:58:34 DEBUG }, 22:58:34 DEBUG 22:58:34 DEBUG "value": [ 22:58:34 DEBUG }, 22:58:34 DEBUG }, 22:58:34 DEBUG "metric": { 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG "value": [ 22:58:34 DEBUG }, 22:58:34 DEBUG }, 22:58:34 DEBUG }, 22:58:34 DEBUG }, 22:58:34 DEBUG }, 22:58:34 DEBUG }, 22:58:34 DEBUG 22:58:34 DEBUG "pod": "ds-idrepo-0" 22:58:34 DEBUG }, 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "instance": "10.142.0.107:9100" 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG }, 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG }, 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG "value": [ 22:58:34 DEBUG [...] 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG "value": [ 22:58:34 DEBUG [...] 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG 22:58:34 DEBUG }, 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:58:34 DEBUG 22:58:34 DEBUG "pod": "ds-cts-2" 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG "pod": "ds-idrepo-2" 22:58:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:58:34 DEBUG }, 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG [...] 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG }, 22:58:34 DEBUG 22:58:34 DEBUG }, 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG }, 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG "pod": "ds-idrepo-2" 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG 22:58:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "pod": "ds-idrepo-2" 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "1812.2666666666667" 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG }, 22:58:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:58:34 DEBUG "pod": "ds-cts-2" 22:58:34 DEBUG }, 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG "0" 22:58:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:58:34 DEBUG }, 22:58:34 DEBUG "value": [ 22:58:34 DEBUG }, 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:58:34 DEBUG 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG ] 22:58:34 DEBUG "instance": "10.142.0.43:9100" 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "pod": "ds-cts-2" 22:58:34 DEBUG }, 22:58:34 DEBUG }, 22:58:34 DEBUG "0" 22:58:34 DEBUG 22:58:34 DEBUG 22:58:34 DEBUG ] 22:58:34 DEBUG }, 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "0" 22:58:34 DEBUG }, 22:58:34 DEBUG "instance": "10.142.0.43:9100" 22:58:34 DEBUG "0" 22:58:34 DEBUG } 22:58:34 DEBUG }, 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG }, 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:58:34 DEBUG 22:58:34 DEBUG } 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "0" 22:58:34 DEBUG "0" 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG ] 22:58:34 DEBUG "value": [ 22:58:34 DEBUG }, 22:58:34 DEBUG ] 22:58:34 DEBUG ] 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "328441.5" 22:58:34 DEBUG "value": [ 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG "0.00015262666666666664" 22:58:34 DEBUG ] 22:58:34 DEBUG }, 22:58:34 DEBUG "instance": "10.142.0.43:9100" 22:58:34 DEBUG }, 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG ] 22:58:34 DEBUG ] 22:58:34 DEBUG "0" 22:58:34 DEBUG } 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG "value": [ 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG ] 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG "6.560686097313327" 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG "value": [ 22:58:34 DEBUG "value": [ 22:58:34 DEBUG ] 22:58:34 DEBUG "0.018" 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG ] 22:58:34 DEBUG "1811.7666666666667" 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG "0.050999999999999775" 22:58:34 DEBUG } 22:58:34 DEBUG "0" 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG 1713391114, 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG "195795059712" 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG ] 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG "0" 22:58:34 DEBUG "60637346.766666666" 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG ] 22:58:34 DEBUG ] 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:58:34 DEBUG } 22:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391172" 22:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391172" 22:59:32 ERROR [http_cmd]: ERROR 22:59:32 ERROR --- status code --- 22:59:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:59:32 ERROR --- http response --- 22:59:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:59:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:59:32 INFO [http_cmd]: http status code OK 22:59:32 DEBUG --- status code --- 22:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:32 DEBUG --- http response --- 22:59:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:59:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391173" 22:59:33 INFO [http_cmd]: http status code OK 22:59:33 DEBUG --- status code --- 22:59:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:33 DEBUG --- http response --- 22:59:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:33 DEBUG ----- output ----- 22:59:33 DEBUG { 22:59:33 DEBUG "status": "success", 22:59:33 DEBUG "data": { 22:59:33 DEBUG "resultType": "vector", 22:59:33 DEBUG "result": [ 22:59:33 DEBUG { 22:59:33 DEBUG "metric": { 22:59:33 DEBUG "pod": "ds-idrepo-1" 22:59:33 DEBUG }, 22:59:33 DEBUG "value": [ 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG [...] 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:59:33 DEBUG }, 22:59:33 DEBUG "value": [ 22:59:33 DEBUG 1713391173, 22:59:33 DEBUG "0" 22:59:33 DEBUG ] 22:59:33 DEBUG } 22:59:33 DEBUG ] 22:59:33 DEBUG } 22:59:33 DEBUG } 22:59:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391173" 22:59:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391173" 22:59:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391173" 22:59:33 INFO [http_cmd]: http status code OK 22:59:33 DEBUG --- status code --- 22:59:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:33 DEBUG --- http response --- 22:59:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:33 DEBUG ----- output ----- 22:59:33 DEBUG { 22:59:33 DEBUG "status": "success", 22:59:33 DEBUG "data": { 22:59:33 DEBUG "resultType": "vector", 22:59:33 DEBUG "result": [ 22:59:33 DEBUG { 22:59:33 DEBUG "metric": { 22:59:33 DEBUG "instance": "10.142.0.107:9100" 22:59:33 DEBUG }, 22:59:33 DEBUG "value": [ 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG [...] 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG "instance": "10.142.0.43:9100" 22:59:33 DEBUG }, 22:59:33 DEBUG "value": [ 22:59:33 DEBUG 1713391173, 22:59:33 DEBUG "0.06066666666666738" 22:59:33 DEBUG ] 22:59:33 DEBUG } 22:59:33 DEBUG ] 22:59:33 DEBUG } 22:59:33 DEBUG } 22:59:33 INFO [http_cmd]: http status code OK 22:59:33 DEBUG --- status code --- 22:59:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:33 DEBUG --- http response --- 22:59:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391173" 22:59:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:33 DEBUG ----- output ----- 22:59:33 DEBUG { 22:59:33 DEBUG "status": "success", 22:59:33 DEBUG "data": { 22:59:33 DEBUG "resultType": "vector", 22:59:33 DEBUG "result": [ 22:59:33 DEBUG { 22:59:33 DEBUG "metric": { 22:59:33 DEBUG "pod": "am-b59c765b7-ch5q4" 22:59:33 DEBUG }, 22:59:33 DEBUG "value": [ 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG [...] 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG "pod": "am-b59c765b7-bgsxp" 22:59:33 DEBUG }, 22:59:33 DEBUG "value": [ 22:59:33 DEBUG 1713391173, 22:59:33 DEBUG "1805.2" 22:59:33 DEBUG ] 22:59:33 DEBUG } 22:59:33 DEBUG ] 22:59:33 DEBUG } 22:59:33 DEBUG } 22:59:33 INFO [http_cmd]: http status code OK 22:59:33 DEBUG --- status code --- 22:59:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:33 DEBUG --- http response --- 22:59:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:33 DEBUG ----- output ----- 22:59:33 DEBUG { 22:59:33 DEBUG "status": "success", 22:59:33 DEBUG "data": { 22:59:33 DEBUG "resultType": "vector", 22:59:33 DEBUG "result": [ 22:59:33 DEBUG { 22:59:33 DEBUG "metric": { 22:59:33 DEBUG "instance": "10.142.0.107:9100" 22:59:33 DEBUG }, 22:59:33 DEBUG "value": [ 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG [...] 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG "instance": "10.142.0.43:9100" 22:59:33 DEBUG }, 22:59:33 DEBUG "value": [ 22:59:33 DEBUG 1713391173, 22:59:33 DEBUG "60654160.766666666" 22:59:33 DEBUG ] 22:59:33 DEBUG } 22:59:33 DEBUG ] 22:59:33 DEBUG } 22:59:33 DEBUG } 22:59:33 INFO [http_cmd]: http status code OK 22:59:33 DEBUG --- status code --- 22:59:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:33 DEBUG --- http response --- 22:59:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:33 DEBUG ----- output ----- 22:59:33 DEBUG { 22:59:33 DEBUG "status": "success", 22:59:33 DEBUG "data": { 22:59:33 DEBUG "resultType": "vector", 22:59:33 DEBUG "result": [ 22:59:33 DEBUG { 22:59:33 DEBUG "metric": { 22:59:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:59:33 DEBUG }, 22:59:33 DEBUG "value": [ 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG [...] 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG 22:59:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:59:33 DEBUG }, 22:59:33 DEBUG "value": [ 22:59:33 DEBUG 1713391173, 22:59:33 DEBUG "2252394496" 22:59:33 DEBUG ] 22:59:33 DEBUG } 22:59:33 DEBUG ] 22:59:33 DEBUG } 22:59:33 DEBUG } 22:59:33 INFO 22:59:33 INFO [loop_until]: kubectl --namespace=xlou top pods 22:59:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:59:33 INFO [loop_until]: OK (rc = 0) 22:59:33 DEBUG --- stdout --- 22:59:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11683m 8915Mi am-b59c765b7-bgsxp 10306m 8771Mi am-b59c765b7-ch5q4 12076m 8536Mi ds-cts-0 12417m 7895Mi ds-cts-1 11481m 7723Mi ds-cts-2 13022m 7970Mi ds-idrepo-0 5216m 3220Mi ds-idrepo-1 5233m 2854Mi ds-idrepo-2 5160m 2932Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 5m 1009Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6659m 2059Mi 22:59:33 DEBUG --- stderr --- 22:59:33 DEBUG 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG { 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG "data": { 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "result": [ 22:59:34 DEBUG { 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "pod": "ds-cts-0" 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG "pod": "ds-cts-2" 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG "29.933333333333334" 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG { 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG "data": { 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "result": [ 22:59:34 DEBUG { 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG "1821.5" 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 INFO 22:59:34 INFO [loop_until]: kubectl --namespace=xlou top node 22:59:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG --- http response --- 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 DEBUG --- status code --- 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391174" 22:59:34 DEBUG --- status code --- 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391174" 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391174" 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG ----- output ----- 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391174" 22:59:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391174" 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG --- status code --- 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG { 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG --- status code --- 22:59:34 INFO [loop_until]: OK (rc = 0) 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG --- status code --- 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:34 DEBUG --- stdout --- 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG "data": { 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG --- http response --- 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG ----- output ----- 22:59:34 INFO [http_cmd]: http status code OK 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 5437Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5249m 17% 6149Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11943m 40% 11120Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 2991Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12103m 40% 10119Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 4974Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11868m 39% 11528Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28114m 94% 2911Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11689m 39% 10618Mi 9% gke-xlou-cdm-ds-0f601751-j488 5528m 18% 5507Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11714m 39% 11291Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5848m 19% 5963Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12862m 43% 9656Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6733m 42% 5842Mi 9% 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG { 22:59:34 DEBUG --- status code --- 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG --- stderr --- 22:59:34 DEBUG { 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:34 DEBUG "result": [ 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG { 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG { 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG { 22:59:34 DEBUG { 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG "data": { 22:59:34 DEBUG --- http response --- 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:34 DEBUG "data": { 22:59:34 DEBUG { 22:59:34 DEBUG { 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG { 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG { 22:59:34 DEBUG "metric": { 22:59:34 DEBUG { 22:59:34 DEBUG { 22:59:34 DEBUG { 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG { 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG "data": { 22:59:34 DEBUG { 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG "data": { 22:59:34 DEBUG "data": { 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG "result": [ 22:59:34 DEBUG ----- output ----- 22:59:34 DEBUG { 22:59:34 DEBUG "result": [ 22:59:34 DEBUG "data": { 22:59:34 DEBUG "data": { 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG "data": { 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG "data": { 22:59:34 DEBUG }, 22:59:34 DEBUG "data": { 22:59:34 DEBUG "data": { 22:59:34 DEBUG "data": { 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG { 22:59:34 DEBUG { 22:59:34 DEBUG { 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG { 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "data": { 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "result": [ 22:59:34 DEBUG "data": { 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "value": [ 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "result": [ 22:59:34 DEBUG "result": [ 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "status": "success", 22:59:34 DEBUG "data": { 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "result": [ 22:59:34 DEBUG "result": [ 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "result": [ 22:59:34 DEBUG { 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "result": [ 22:59:34 DEBUG 22:59:34 DEBUG "result": [ 22:59:34 DEBUG "result": [ 22:59:34 DEBUG "result": [ 22:59:34 DEBUG { 22:59:34 DEBUG { 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "pod": "ds-idrepo-0" 22:59:34 DEBUG "data": { 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "pod": "ds-idrepo-0" 22:59:34 DEBUG { 22:59:34 DEBUG { 22:59:34 DEBUG "result": [ 22:59:34 DEBUG { 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "result": [ 22:59:34 DEBUG { 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG { 22:59:34 DEBUG { 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "pod": "ds-idrepo-1" 22:59:34 DEBUG }, 22:59:34 DEBUG "data": { 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "result": [ 22:59:34 DEBUG }, 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "metric": { 22:59:34 DEBUG { 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:59:34 DEBUG { 22:59:34 DEBUG "metric": { 22:59:34 DEBUG { 22:59:34 DEBUG [...] 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "pod": "ds-cts-0" 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG "resultType": "vector", 22:59:34 DEBUG "result": [ 22:59:34 DEBUG { 22:59:34 DEBUG "value": [ 22:59:34 DEBUG "pod": "ds-idrepo-0" 22:59:34 DEBUG "pod": "ds-cts-0" 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:59:34 DEBUG }, 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:59:34 DEBUG "metric": { 22:59:34 DEBUG 22:59:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG "result": [ 22:59:34 DEBUG { 22:59:34 DEBUG "metric": { 22:59:34 DEBUG 22:59:34 DEBUG }, 22:59:34 DEBUG }, 22:59:34 DEBUG "instance": "10.142.0.107:9100" 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG "instance": "10.142.0.107:9100" 22:59:34 DEBUG }, 22:59:34 DEBUG "pod": "ds-idrepo-1" 22:59:34 DEBUG 22:59:34 DEBUG "pod": "ds-idrepo-0" 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG { 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "pod": "am-b59c765b7-ch5q4" 22:59:34 DEBUG 22:59:34 DEBUG "value": [ 22:59:34 DEBUG "value": [ 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG "metric": { 22:59:34 DEBUG "instance": "10.142.0.107:9100" 22:59:34 DEBUG }, 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG }, 22:59:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG [...] 22:59:34 DEBUG "pod": "ds-idrepo-1" 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG "value": [ 22:59:34 DEBUG }, 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG }, 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG "pod": "ds-cts-1" 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG "1804.9" 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:59:34 DEBUG }, 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG "pod": "ds-idrepo-2" 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG ] 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG }, 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG }, 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG } 22:59:34 DEBUG 22:59:34 DEBUG "pod": "ds-cts-2" 22:59:34 DEBUG "pod": "ds-idrepo-2" 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG "value": [ 22:59:34 DEBUG "pod": "ds-idrepo-2" 22:59:34 DEBUG "pod": "ds-cts-2" 22:59:34 DEBUG 22:59:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:59:34 DEBUG 22:59:34 DEBUG "value": [ 22:59:34 DEBUG "instance": "10.142.0.43:9100" 22:59:34 DEBUG 22:59:34 DEBUG ] 22:59:34 DEBUG }, 22:59:34 DEBUG 22:59:34 DEBUG }, 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG "50217106.62234723" 22:59:34 DEBUG [...] 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG }, 22:59:34 DEBUG }, 22:59:34 DEBUG "instance": "10.142.0.43:9100" 22:59:34 DEBUG }, 22:59:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG }, 22:59:34 DEBUG 22:59:34 DEBUG } 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 22:59:34 DEBUG "value": [ 22:59:34 DEBUG "6.608456099438086" 22:59:34 DEBUG ] 22:59:34 DEBUG 22:59:34 DEBUG 22:59:34 DEBUG "pod": "am-b59c765b7-bgsxp" 22:59:34 DEBUG "0" 22:59:34 DEBUG "value": [ 22:59:34 DEBUG "value": [ 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG }, 22:59:34 DEBUG "1805.6" 22:59:34 DEBUG "value": [ 22:59:34 DEBUG "pod": "ds-cts-1" 22:59:34 DEBUG } 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG "pod": "ds-cts-2" 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG 22:59:34 DEBUG "instance": "10.142.0.43:9100" 22:59:34 DEBUG }, 22:59:34 DEBUG ] 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG ] 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG }, 22:59:34 DEBUG "0" 22:59:34 DEBUG }, 22:59:34 DEBUG "0" 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG 22:59:34 DEBUG }, 22:59:34 DEBUG "value": [ 22:59:34 DEBUG } 22:59:34 DEBUG "0" 22:59:34 DEBUG "0" 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG "0" 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG } 22:59:34 DEBUG "0.04166666666666673" 22:59:34 DEBUG "value": [ 22:59:34 DEBUG ] 22:59:34 DEBUG "value": [ 22:59:34 DEBUG ] 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:59:34 DEBUG "value": [ 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG ] 22:59:34 DEBUG ] 22:59:34 DEBUG ] 22:59:34 DEBUG "198759640064" 22:59:34 DEBUG ] 22:59:34 DEBUG "328120" 22:59:34 DEBUG ] 22:59:34 DEBUG ] 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG } 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG }, 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG "0.018" 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG "0" 22:59:34 DEBUG ] 22:59:34 DEBUG "0" 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG "value": [ 22:59:34 DEBUG "60654197.266666666" 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG 1713391174, 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG "0" 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG } 22:59:34 DEBUG ] 22:59:34 DEBUG } 22:59:34 DEBUG } 23:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391232" 23:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391232" 23:00:32 ERROR [http_cmd]: ERROR 23:00:32 ERROR --- status code --- 23:00:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:00:32 ERROR --- http response --- 23:00:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:00:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:00:32 INFO [http_cmd]: http status code OK 23:00:32 DEBUG --- status code --- 23:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:32 DEBUG --- http response --- 23:00:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:00:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391233" 23:00:33 INFO [http_cmd]: http status code OK 23:00:33 DEBUG --- status code --- 23:00:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:33 DEBUG --- http response --- 23:00:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:00:33 DEBUG ----- output ----- 23:00:33 DEBUG { 23:00:33 DEBUG "status": "success", 23:00:33 DEBUG "data": { 23:00:33 DEBUG "resultType": "vector", 23:00:33 DEBUG "result": [ 23:00:33 DEBUG { 23:00:33 DEBUG "metric": { 23:00:33 DEBUG "pod": "ds-idrepo-1" 23:00:33 DEBUG }, 23:00:33 DEBUG "value": [ 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG [...] 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG "pod": "ds-idrepo-2" 23:00:33 DEBUG }, 23:00:33 DEBUG "value": [ 23:00:33 DEBUG 1713391233, 23:00:33 DEBUG "0" 23:00:33 DEBUG ] 23:00:33 DEBUG } 23:00:33 DEBUG ] 23:00:33 DEBUG } 23:00:33 DEBUG } 23:00:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391233" 23:00:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391233" 23:00:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391233" 23:00:33 INFO [http_cmd]: http status code OK 23:00:33 INFO [http_cmd]: http status code OK 23:00:33 DEBUG --- status code --- 23:00:33 DEBUG --- status code --- 23:00:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:33 DEBUG --- http response --- 23:00:33 DEBUG --- http response --- 23:00:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:33 DEBUG ----- output ----- 23:00:33 DEBUG ----- output ----- 23:00:33 DEBUG { 23:00:33 DEBUG { 23:00:33 DEBUG "status": "success", 23:00:33 DEBUG "status": "success", 23:00:33 DEBUG "data": { 23:00:33 DEBUG "data": { 23:00:33 DEBUG "resultType": "vector", 23:00:33 DEBUG "resultType": "vector", 23:00:33 DEBUG "result": [ 23:00:33 DEBUG "result": [ 23:00:33 DEBUG { 23:00:33 DEBUG { 23:00:33 DEBUG "metric": { 23:00:33 DEBUG "metric": { 23:00:33 DEBUG "pod": "am-b59c765b7-ch5q4" 23:00:33 DEBUG "instance": "10.142.0.107:9100" 23:00:33 DEBUG }, 23:00:33 DEBUG }, 23:00:33 DEBUG "value": [ 23:00:33 DEBUG "value": [ 23:00:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391233" 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG [...] 23:00:33 DEBUG [...] 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG "pod": "am-b59c765b7-bgsxp" 23:00:33 DEBUG "instance": "10.142.0.43:9100" 23:00:33 DEBUG }, 23:00:33 DEBUG }, 23:00:33 DEBUG "value": [ 23:00:33 DEBUG "value": [ 23:00:33 DEBUG 1713391233, 23:00:33 DEBUG 1713391233, 23:00:33 DEBUG "1826.9333333333334" 23:00:33 DEBUG "0.15926666666666545" 23:00:33 DEBUG ] 23:00:33 DEBUG ] 23:00:33 DEBUG } 23:00:33 DEBUG } 23:00:33 DEBUG ] 23:00:33 INFO [http_cmd]: http status code OK 23:00:33 DEBUG ] 23:00:33 DEBUG } 23:00:33 DEBUG --- status code --- 23:00:33 DEBUG } 23:00:33 DEBUG } 23:00:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:33 DEBUG } 23:00:33 DEBUG --- http response --- 23:00:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:33 DEBUG ----- output ----- 23:00:33 DEBUG { 23:00:33 DEBUG "status": "success", 23:00:33 DEBUG "data": { 23:00:33 DEBUG "resultType": "vector", 23:00:33 DEBUG "result": [ 23:00:33 DEBUG { 23:00:33 DEBUG "metric": { 23:00:33 DEBUG "instance": "10.142.0.107:9100" 23:00:33 DEBUG }, 23:00:33 DEBUG "value": [ 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG [...] 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG "instance": "10.142.0.43:9100" 23:00:33 DEBUG }, 23:00:33 DEBUG "value": [ 23:00:33 DEBUG 1713391233, 23:00:33 DEBUG "60928595.5" 23:00:33 DEBUG ] 23:00:33 DEBUG } 23:00:33 DEBUG ] 23:00:33 DEBUG } 23:00:33 DEBUG } 23:00:33 INFO [http_cmd]: http status code OK 23:00:33 DEBUG --- status code --- 23:00:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:33 DEBUG --- http response --- 23:00:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:33 DEBUG ----- output ----- 23:00:33 DEBUG { 23:00:33 DEBUG "status": "success", 23:00:33 DEBUG "data": { 23:00:33 DEBUG "resultType": "vector", 23:00:33 DEBUG "result": [ 23:00:33 DEBUG { 23:00:33 DEBUG "metric": { 23:00:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:00:33 DEBUG }, 23:00:33 DEBUG "value": [ 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG [...] 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG 23:00:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:00:33 DEBUG }, 23:00:33 DEBUG "value": [ 23:00:33 DEBUG 1713391233, 23:00:33 DEBUG "2257997824" 23:00:33 DEBUG ] 23:00:33 DEBUG } 23:00:33 DEBUG ] 23:00:33 DEBUG } 23:00:33 DEBUG } 23:00:33 INFO 23:00:33 INFO [loop_until]: kubectl --namespace=xlou top pods 23:00:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:00:33 INFO [loop_until]: OK (rc = 0) 23:00:33 DEBUG --- stdout --- 23:00:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12126m 8917Mi am-b59c765b7-bgsxp 10938m 8771Mi am-b59c765b7-ch5q4 11869m 8536Mi ds-cts-0 12786m 7876Mi ds-cts-1 11708m 7730Mi ds-cts-2 13138m 7969Mi ds-idrepo-0 5301m 3220Mi ds-idrepo-1 5294m 2855Mi ds-idrepo-2 5253m 2931Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 3m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6619m 2062Mi 23:00:33 DEBUG --- stderr --- 23:00:33 DEBUG 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG { 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG "data": { 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "result": [ 23:00:34 DEBUG { 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "pod": "ds-cts-0" 23:00:34 DEBUG }, 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG [...] 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG "pod": "ds-cts-2" 23:00:34 DEBUG }, 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG "30" 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG { 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG "data": { 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "result": [ 23:00:34 DEBUG { 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:00:34 DEBUG }, 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG [...] 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:00:34 DEBUG }, 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG "1843.5666666666666" 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 INFO 23:00:34 INFO [loop_until]: kubectl --namespace=xlou top node 23:00:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391234" 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG --- status code --- 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:34 INFO [http_cmd]: http status code OK 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG --- status code --- 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG { 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG { 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG "data": { 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:34 INFO [loop_until]: OK (rc = 0) 23:00:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG { 23:00:34 DEBUG { 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG { 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG { 23:00:34 DEBUG { 23:00:34 DEBUG { 23:00:34 DEBUG { 23:00:34 DEBUG { 23:00:34 DEBUG { 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG "data": { 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "result": [ 23:00:34 DEBUG --- stdout --- 23:00:34 DEBUG --- http response --- 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG { 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG "data": { 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG { 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG { 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG { 23:00:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 5449Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5141m 17% 6143Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11923m 39% 11117Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 2994Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12356m 41% 10128Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 164m 0% 4938Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11940m 39% 11520Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28037m 93% 2925Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11906m 39% 10614Mi 9% gke-xlou-cdm-ds-0f601751-j488 5257m 17% 5504Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11689m 39% 11286Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5865m 19% 5967Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12974m 43% 9657Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6765m 42% 5845Mi 9% 23:00:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:00:34 DEBUG "data": { 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "data": { 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG "data": { 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG { 23:00:34 DEBUG "data": { 23:00:34 DEBUG "data": { 23:00:34 DEBUG "data": { 23:00:34 DEBUG "data": { 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG "result": [ 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG { 23:00:34 DEBUG "metric": { 23:00:34 DEBUG --- stderr --- 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG "result": [ 23:00:34 DEBUG ----- output ----- 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "data": { 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "result": [ 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG "data": { 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "data": { 23:00:34 DEBUG { 23:00:34 DEBUG "data": { 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG "pod": "ds-idrepo-0" 23:00:34 DEBUG { 23:00:34 DEBUG 23:00:34 DEBUG { 23:00:34 DEBUG { 23:00:34 DEBUG "result": [ 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "result": [ 23:00:34 DEBUG { 23:00:34 DEBUG "data": { 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "result": [ 23:00:34 DEBUG "result": [ 23:00:34 DEBUG "result": [ 23:00:34 DEBUG "result": [ 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "data": { 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "status": "success", 23:00:34 DEBUG { 23:00:34 DEBUG "result": [ 23:00:34 DEBUG { 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "result": [ 23:00:34 DEBUG { 23:00:34 DEBUG { 23:00:34 DEBUG { 23:00:34 DEBUG { 23:00:34 DEBUG "result": [ 23:00:34 DEBUG "pod": "ds-cts-0" 23:00:34 DEBUG "result": [ 23:00:34 DEBUG "result": [ 23:00:34 DEBUG }, 23:00:34 DEBUG "data": { 23:00:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:00:34 DEBUG "data": { 23:00:34 DEBUG "metric": { 23:00:34 DEBUG { 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "pod": "ds-idrepo-1" 23:00:34 DEBUG "result": [ 23:00:34 DEBUG { 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "metric": { 23:00:34 DEBUG { 23:00:34 DEBUG }, 23:00:34 DEBUG { 23:00:34 DEBUG { 23:00:34 DEBUG "value": [ 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG }, 23:00:34 DEBUG "resultType": "vector", 23:00:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "pod": "ds-idrepo-0" 23:00:34 DEBUG }, 23:00:34 DEBUG { 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:00:34 DEBUG }, 23:00:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:00:34 DEBUG "instance": "10.142.0.107:9100" 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "value": [ 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "metric": { 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG "value": [ 23:00:34 DEBUG "result": [ 23:00:34 DEBUG }, 23:00:34 DEBUG "value": [ 23:00:34 DEBUG }, 23:00:34 DEBUG "value": [ 23:00:34 DEBUG "metric": { 23:00:34 DEBUG "pod": "ds-cts-0" 23:00:34 DEBUG "value": [ 23:00:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:00:34 DEBUG }, 23:00:34 DEBUG }, 23:00:34 DEBUG "pod": "ds-idrepo-1" 23:00:34 DEBUG 23:00:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:00:34 DEBUG "instance": "10.142.0.107:9100" 23:00:34 DEBUG 23:00:34 DEBUG "result": [ 23:00:34 DEBUG 23:00:34 DEBUG { 23:00:34 DEBUG "pod": "ds-idrepo-0" 23:00:34 DEBUG 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 23:00:34 DEBUG "pod": "ds-idrepo-1" 23:00:34 DEBUG }, 23:00:34 DEBUG 23:00:34 DEBUG }, 23:00:34 DEBUG "value": [ 23:00:34 DEBUG "value": [ 23:00:34 DEBUG }, 23:00:34 DEBUG 23:00:34 DEBUG }, 23:00:34 DEBUG "value": [ 23:00:34 DEBUG [...] 23:00:34 DEBUG { 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG }, 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG }, 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 23:00:34 DEBUG }, 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG "metric": { 23:00:34 DEBUG [...] 23:00:34 DEBUG "value": [ 23:00:34 DEBUG "metric": { 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 23:00:34 DEBUG "value": [ 23:00:34 DEBUG [...] 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG [...] 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG "instance": "10.142.0.107:9100" 23:00:34 DEBUG [...] 23:00:34 DEBUG [...] 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG "pod": "ds-idrepo-2" 23:00:34 DEBUG "pod": "ds-idrepo-0" 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG }, 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG [...] 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG [...] 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG [...] 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG [...] 23:00:34 DEBUG }, 23:00:34 DEBUG }, 23:00:34 DEBUG 23:00:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:00:34 DEBUG }, 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG [...] 23:00:34 DEBUG "pod": "ds-cts-2" 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG "value": [ 23:00:34 DEBUG "value": [ 23:00:34 DEBUG }, 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG [...] 23:00:34 DEBUG [...] 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG }, 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG 23:00:34 DEBUG "value": [ 23:00:34 DEBUG [...] 23:00:34 DEBUG 23:00:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:00:34 DEBUG "pod": "ds-idrepo-2" 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG [...] 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG "0" 23:00:34 DEBUG "0" 23:00:34 DEBUG 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG }, 23:00:34 DEBUG }, 23:00:34 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 23:00:34 DEBUG "pod": "ds-idrepo-2" 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG "1825.6" 23:00:34 DEBUG 23:00:34 DEBUG "instance": "10.142.0.43:9100" 23:00:34 DEBUG 23:00:34 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 23:00:34 DEBUG 23:00:34 DEBUG [...] 23:00:34 DEBUG ] 23:00:34 DEBUG 23:00:34 DEBUG "0.018" 23:00:34 DEBUG 23:00:34 DEBUG [...] 23:00:34 DEBUG "value": [ 23:00:34 DEBUG "value": [ 23:00:34 DEBUG }, 23:00:34 DEBUG }, 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG ] 23:00:34 DEBUG 23:00:34 DEBUG }, 23:00:34 DEBUG }, 23:00:34 DEBUG ] 23:00:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:00:34 DEBUG 23:00:34 DEBUG } 23:00:34 DEBUG [...] 23:00:34 DEBUG ] 23:00:34 DEBUG 23:00:34 DEBUG 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG "0" 23:00:34 DEBUG "value": [ 23:00:34 DEBUG "pod": "ds-cts-2" 23:00:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:00:34 DEBUG } 23:00:34 DEBUG 23:00:34 DEBUG "value": [ 23:00:34 DEBUG "value": [ 23:00:34 DEBUG } 23:00:34 DEBUG }, 23:00:34 DEBUG 23:00:34 DEBUG ] 23:00:34 DEBUG 23:00:34 DEBUG } 23:00:34 DEBUG "pod": "ds-idrepo-2" 23:00:34 DEBUG 23:00:34 DEBUG "1827.4" 23:00:34 DEBUG "value": [ 23:00:34 DEBUG ] 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG }, 23:00:34 DEBUG }, 23:00:34 DEBUG ] 23:00:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG ] 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 23:00:34 DEBUG } 23:00:34 DEBUG 23:00:34 DEBUG ] 23:00:34 DEBUG }, 23:00:34 DEBUG 23:00:34 DEBUG "instance": "10.142.0.43:9100" 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG } 23:00:34 DEBUG "0" 23:00:34 DEBUG ] 23:00:34 DEBUG "value": [ 23:00:34 DEBUG } 23:00:34 DEBUG }, 23:00:34 DEBUG "201612106752" 23:00:34 DEBUG "0" 23:00:34 DEBUG } 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG "instance": "10.142.0.43:9100" 23:00:34 DEBUG } 23:00:34 DEBUG 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG }, 23:00:34 DEBUG ] 23:00:34 DEBUG "0" 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG "value": [ 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG } 23:00:34 DEBUG "value": [ 23:00:34 DEBUG ] 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG "328083" 23:00:34 DEBUG }, 23:00:34 DEBUG "pod": "ds-cts-2" 23:00:34 DEBUG "value": [ 23:00:34 DEBUG "value": [ 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG "0" 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG "value": [ 23:00:34 DEBUG }, 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG "0.00002266" 23:00:34 DEBUG ] 23:00:34 DEBUG "6.595672949784568" 23:00:34 DEBUG ] 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG "value": [ 23:00:34 DEBUG "0" 23:00:34 DEBUG "0.07300000000000005" 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG "60927722.400000006" 23:00:34 DEBUG 1713391234, 23:00:34 DEBUG ] 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG ] 23:00:34 DEBUG "0" 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG ] 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG ] 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:00:34 DEBUG } 23:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391292" 23:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391292" 23:01:33 ERROR [http_cmd]: ERROR 23:01:33 ERROR --- status code --- 23:01:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:01:33 ERROR --- http response --- 23:01:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:01:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:01:33 INFO [http_cmd]: http status code OK 23:01:33 DEBUG --- status code --- 23:01:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:33 DEBUG --- http response --- 23:01:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:01:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391293" 23:01:33 INFO [http_cmd]: http status code OK 23:01:33 DEBUG --- status code --- 23:01:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:33 DEBUG --- http response --- 23:01:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:33 DEBUG ----- output ----- 23:01:33 DEBUG { 23:01:33 DEBUG "status": "success", 23:01:33 DEBUG "data": { 23:01:33 DEBUG "resultType": "vector", 23:01:33 DEBUG "result": [ 23:01:33 DEBUG { 23:01:33 DEBUG "metric": { 23:01:33 DEBUG "pod": "ds-idrepo-0" 23:01:33 DEBUG }, 23:01:33 DEBUG "value": [ 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG [...] 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:01:33 DEBUG }, 23:01:33 DEBUG "value": [ 23:01:33 DEBUG 1713391293, 23:01:33 DEBUG "0.000021614343420026906" 23:01:33 DEBUG ] 23:01:33 DEBUG } 23:01:33 DEBUG ] 23:01:33 DEBUG } 23:01:33 DEBUG } 23:01:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391293" 23:01:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391293" 23:01:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391293" 23:01:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391293" 23:01:33 INFO [http_cmd]: http status code OK 23:01:33 DEBUG --- status code --- 23:01:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:33 DEBUG --- http response --- 23:01:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:33 DEBUG ----- output ----- 23:01:33 DEBUG { 23:01:33 DEBUG "status": "success", 23:01:33 DEBUG "data": { 23:01:33 DEBUG "resultType": "vector", 23:01:33 DEBUG "result": [ 23:01:33 DEBUG { 23:01:33 DEBUG "metric": { 23:01:33 DEBUG "instance": "10.142.0.107:9100" 23:01:33 DEBUG }, 23:01:33 DEBUG "value": [ 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG [...] 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG "instance": "10.142.0.43:9100" 23:01:33 DEBUG }, 23:01:33 DEBUG "value": [ 23:01:33 DEBUG 1713391293, 23:01:33 DEBUG "0.08609999999999994" 23:01:33 DEBUG ] 23:01:33 DEBUG } 23:01:33 DEBUG ] 23:01:33 DEBUG } 23:01:33 DEBUG } 23:01:33 INFO [http_cmd]: http status code OK 23:01:33 DEBUG --- status code --- 23:01:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:33 DEBUG --- http response --- 23:01:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:33 DEBUG ----- output ----- 23:01:33 DEBUG { 23:01:33 DEBUG "status": "success", 23:01:33 DEBUG "data": { 23:01:33 DEBUG "resultType": "vector", 23:01:33 DEBUG "result": [ 23:01:33 DEBUG { 23:01:33 DEBUG "metric": { 23:01:33 DEBUG "pod": "am-b59c765b7-ch5q4" 23:01:33 DEBUG }, 23:01:33 DEBUG "value": [ 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG [...] 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG "pod": "am-b59c765b7-bgsxp" 23:01:33 DEBUG }, 23:01:33 DEBUG "value": [ 23:01:33 DEBUG 1713391293, 23:01:33 DEBUG "1827.4333333333334" 23:01:33 DEBUG ] 23:01:33 DEBUG } 23:01:33 DEBUG ] 23:01:33 DEBUG } 23:01:33 DEBUG } 23:01:33 INFO [http_cmd]: http status code OK 23:01:33 DEBUG --- status code --- 23:01:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:33 DEBUG --- http response --- 23:01:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:33 INFO [http_cmd]: http status code OK 23:01:33 DEBUG ----- output ----- 23:01:33 DEBUG --- status code --- 23:01:33 DEBUG { 23:01:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:33 DEBUG "status": "success", 23:01:33 DEBUG --- http response --- 23:01:33 DEBUG "data": { 23:01:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:33 DEBUG "resultType": "vector", 23:01:33 DEBUG ----- output ----- 23:01:33 DEBUG "result": [ 23:01:33 DEBUG { 23:01:33 DEBUG { 23:01:33 DEBUG "status": "success", 23:01:33 DEBUG "metric": { 23:01:33 DEBUG "data": { 23:01:33 DEBUG "instance": "10.142.0.107:9100" 23:01:33 DEBUG "resultType": "vector", 23:01:33 DEBUG "result": [ 23:01:33 DEBUG }, 23:01:33 DEBUG { 23:01:33 DEBUG "value": [ 23:01:33 DEBUG "metric": { 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:01:33 DEBUG 23:01:33 DEBUG }, 23:01:33 DEBUG "value": [ 23:01:33 DEBUG 23:01:33 DEBUG [...] 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG [...] 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG "instance": "10.142.0.43:9100" 23:01:33 DEBUG 23:01:33 DEBUG 23:01:33 DEBUG }, 23:01:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:01:33 DEBUG "value": [ 23:01:33 DEBUG }, 23:01:33 DEBUG 1713391293, 23:01:33 DEBUG "value": [ 23:01:33 DEBUG "60898461.43333334" 23:01:33 DEBUG 1713391293, 23:01:33 DEBUG ] 23:01:33 DEBUG "2260197376" 23:01:33 DEBUG } 23:01:33 DEBUG ] 23:01:33 DEBUG ] 23:01:33 DEBUG } 23:01:33 DEBUG } 23:01:33 DEBUG ] 23:01:33 DEBUG } 23:01:33 DEBUG } 23:01:33 DEBUG } 23:01:33 INFO 23:01:33 INFO [loop_until]: kubectl --namespace=xlou top pods 23:01:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:01:34 INFO [loop_until]: OK (rc = 0) 23:01:34 DEBUG --- stdout --- 23:01:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11875m 8917Mi am-b59c765b7-bgsxp 11533m 8770Mi am-b59c765b7-ch5q4 11658m 8537Mi ds-cts-0 12636m 7875Mi ds-cts-1 11391m 7747Mi ds-cts-2 12966m 7957Mi ds-idrepo-0 5364m 3220Mi ds-idrepo-1 5274m 2855Mi ds-idrepo-2 5199m 2931Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 5m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6730m 2063Mi 23:01:34 DEBUG --- stderr --- 23:01:34 DEBUG 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG { 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG "data": { 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "result": [ 23:01:34 DEBUG { 23:01:34 DEBUG "metric": { 23:01:34 DEBUG "pod": "ds-cts-0" 23:01:34 DEBUG }, 23:01:34 DEBUG "value": [ 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG "pod": "ds-cts-2" 23:01:34 DEBUG }, 23:01:34 DEBUG "value": [ 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG "29.933333333333334" 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG { 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG "data": { 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "result": [ 23:01:34 DEBUG { 23:01:34 DEBUG "metric": { 23:01:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:01:34 DEBUG }, 23:01:34 DEBUG "value": [ 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:01:34 DEBUG }, 23:01:34 DEBUG "value": [ 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG "1812.0333333333333" 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO 23:01:34 INFO [loop_until]: kubectl --namespace=xlou top node 23:01:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG --- http response --- 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG { 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:34 DEBUG "data": { 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG { 23:01:34 DEBUG "result": [ 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG { 23:01:34 DEBUG "data": { 23:01:34 DEBUG "metric": { 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:01:34 DEBUG "result": [ 23:01:34 DEBUG }, 23:01:34 DEBUG { 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "metric": { 23:01:34 DEBUG 23:01:34 DEBUG "pod": "ds-idrepo-0" 23:01:34 DEBUG 23:01:34 DEBUG }, 23:01:34 DEBUG 23:01:34 DEBUG "value": [ 23:01:34 DEBUG [...] 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:01:34 DEBUG 23:01:34 DEBUG }, 23:01:34 DEBUG 23:01:34 DEBUG "value": [ 23:01:34 DEBUG 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:01:34 DEBUG "1826.7666666666667" 23:01:34 DEBUG }, 23:01:34 DEBUG ] 23:01:34 DEBUG "value": [ 23:01:34 DEBUG } 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG ] 23:01:34 DEBUG "0" 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391294" 23:01:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391294" 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG --- status code --- 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG --- status code --- 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 INFO [loop_until]: OK (rc = 0) 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG --- status code --- 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG --- stdout --- 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 5442Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5242m 17% 6152Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11913m 39% 11120Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 2999Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12380m 41% 10132Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 164m 0% 4956Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11872m 39% 11521Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27925m 93% 2923Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11883m 39% 10618Mi 9% gke-xlou-cdm-ds-0f601751-j488 5508m 18% 5510Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11571m 38% 11293Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5844m 19% 5966Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12980m 43% 9666Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6716m 42% 5847Mi 9% 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:34 INFO [http_cmd]: http status code OK 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG --- stderr --- 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG --- status code --- 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG { 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:01:34 DEBUG { 23:01:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:34 DEBUG { 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:01:34 DEBUG { 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG { 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG { 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG { 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG { 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG --- http response --- 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG { 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG "data": { 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG { 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG { 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG { 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG "data": { 23:01:34 DEBUG "data": { 23:01:34 DEBUG "data": { 23:01:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:34 DEBUG { 23:01:34 DEBUG "data": { 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG "data": { 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG { 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG "data": { 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG "data": { 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG { 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG ----- output ----- 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "data": { 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "result": [ 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG "data": { 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "data": { 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "data": { 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG "result": [ 23:01:34 DEBUG "result": [ 23:01:34 DEBUG "result": [ 23:01:34 DEBUG { 23:01:34 DEBUG "data": { 23:01:34 DEBUG "result": [ 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "result": [ 23:01:34 DEBUG { 23:01:34 DEBUG "data": { 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "result": [ 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "result": [ 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "data": { 23:01:34 DEBUG { 23:01:34 DEBUG { 23:01:34 DEBUG { 23:01:34 DEBUG "status": "success", 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG { 23:01:34 DEBUG "result": [ 23:01:34 DEBUG { 23:01:34 DEBUG "metric": { 23:01:34 DEBUG "pod": "ds-cts-0" 23:01:34 DEBUG }, 23:01:34 DEBUG "result": [ 23:01:34 DEBUG "result": [ 23:01:34 DEBUG { 23:01:34 DEBUG "result": [ 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "metric": { 23:01:34 DEBUG "metric": { 23:01:34 DEBUG "metric": { 23:01:34 DEBUG "data": { 23:01:34 DEBUG "result": [ 23:01:34 DEBUG "metric": { 23:01:34 DEBUG { 23:01:34 DEBUG "metric": { 23:01:34 DEBUG "metric": { 23:01:34 DEBUG "value": [ 23:01:34 DEBUG { 23:01:34 DEBUG { 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG "metric": { 23:01:34 DEBUG { 23:01:34 DEBUG "result": [ 23:01:34 DEBUG "pod": "ds-idrepo-0" 23:01:34 DEBUG "instance": "10.142.0.107:9100" 23:01:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:01:34 DEBUG "resultType": "vector", 23:01:34 DEBUG { 23:01:34 DEBUG "pod": "ds-idrepo-0" 23:01:34 DEBUG { 23:01:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:01:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:01:34 DEBUG 23:01:34 DEBUG "metric": { 23:01:34 DEBUG "metric": { 23:01:34 DEBUG "result": [ 23:01:34 DEBUG "pod": "ds-idrepo-0" 23:01:34 DEBUG "metric": { 23:01:34 DEBUG { 23:01:34 DEBUG }, 23:01:34 DEBUG }, 23:01:34 DEBUG }, 23:01:34 DEBUG "result": [ 23:01:34 DEBUG "metric": { 23:01:34 DEBUG }, 23:01:34 DEBUG "metric": { 23:01:34 DEBUG }, 23:01:34 DEBUG }, 23:01:34 DEBUG 23:01:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:01:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:01:34 DEBUG { 23:01:34 DEBUG }, 23:01:34 DEBUG "pod": "ds-idrepo-0" 23:01:34 DEBUG "metric": { 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "value": [ 23:01:34 DEBUG { 23:01:34 DEBUG "pod": "ds-idrepo-0" 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "pod": "ds-idrepo-0" 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "value": [ 23:01:34 DEBUG 23:01:34 DEBUG }, 23:01:34 DEBUG }, 23:01:34 DEBUG "metric": { 23:01:34 DEBUG "value": [ 23:01:34 DEBUG }, 23:01:34 DEBUG "pod": "ds-cts-0" 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG "metric": { 23:01:34 DEBUG }, 23:01:34 DEBUG 23:01:34 DEBUG }, 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "instance": "10.142.0.107:9100" 23:01:34 DEBUG 23:01:34 DEBUG "value": [ 23:01:34 DEBUG }, 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG "instance": "10.142.0.107:9100" 23:01:34 DEBUG "value": [ 23:01:34 DEBUG 23:01:34 DEBUG "value": [ 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG }, 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG "value": [ 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG }, 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG "value": [ 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG [...] 23:01:34 DEBUG [...] 23:01:34 DEBUG "value": [ 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG [...] 23:01:34 DEBUG 23:01:34 DEBUG "pod": "ds-cts-2" 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG }, 23:01:34 DEBUG [...] 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG "value": [ 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG 23:01:34 DEBUG [...] 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:01:34 DEBUG "instance": "10.142.0.43:9100" 23:01:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:01:34 DEBUG [...] 23:01:34 DEBUG 23:01:34 DEBUG "pod": "ds-idrepo-2" 23:01:34 DEBUG 23:01:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:01:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:01:34 DEBUG 23:01:34 DEBUG "0" 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG "pod": "ds-idrepo-2" 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG }, 23:01:34 DEBUG }, 23:01:34 DEBUG }, 23:01:34 DEBUG 23:01:34 DEBUG 23:01:34 DEBUG }, 23:01:34 DEBUG 23:01:34 DEBUG }, 23:01:34 DEBUG }, 23:01:34 DEBUG 23:01:34 DEBUG ] 23:01:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:01:34 DEBUG 23:01:34 DEBUG }, 23:01:34 DEBUG "pod": "ds-idrepo-2" 23:01:34 DEBUG 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "value": [ 23:01:34 DEBUG 23:01:34 DEBUG "pod": "ds-cts-2" 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:01:34 DEBUG } 23:01:34 DEBUG }, 23:01:34 DEBUG 23:01:34 DEBUG "value": [ 23:01:34 DEBUG }, 23:01:34 DEBUG "pod": "ds-cts-2" 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG 23:01:34 DEBUG }, 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG }, 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG }, 23:01:34 DEBUG ] 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "instance": "10.142.0.43:9100" 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG "value": [ 23:01:34 DEBUG }, 23:01:34 DEBUG "0" 23:01:34 DEBUG "204567614464" 23:01:34 DEBUG "0.018" 23:01:34 DEBUG "instance": "10.142.0.43:9100" 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "0" 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "328305.5" 23:01:34 DEBUG "1827.5" 23:01:34 DEBUG "value": [ 23:01:34 DEBUG } 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG }, 23:01:34 DEBUG "0" 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG "value": [ 23:01:34 DEBUG ] 23:01:34 DEBUG ] 23:01:34 DEBUG ] 23:01:34 DEBUG }, 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG ] 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG ] 23:01:34 DEBUG ] 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG } 23:01:34 DEBUG "6.638048106603872" 23:01:34 DEBUG "value": [ 23:01:34 DEBUG ] 23:01:34 DEBUG "0" 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG "value": [ 23:01:34 DEBUG "0" 23:01:34 DEBUG } 23:01:34 DEBUG "0" 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG "0" 23:01:34 DEBUG ] 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG "0" 23:01:34 DEBUG ] 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG 1713391294, 23:01:34 DEBUG ] 23:01:34 DEBUG ] 23:01:34 DEBUG ] 23:01:34 DEBUG ] 23:01:34 DEBUG ] 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG "0.05500000000000002" 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG "60897367.43333333" 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG ] 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:01:34 DEBUG } 23:02:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391353" 23:02:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391353" 23:02:33 ERROR [http_cmd]: ERROR 23:02:33 ERROR --- status code --- 23:02:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:02:33 ERROR --- http response --- 23:02:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:02:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:02:33 INFO [http_cmd]: http status code OK 23:02:33 DEBUG --- status code --- 23:02:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:33 DEBUG --- http response --- 23:02:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:02:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391353" 23:02:33 INFO [http_cmd]: http status code OK 23:02:33 DEBUG --- status code --- 23:02:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:33 DEBUG --- http response --- 23:02:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:02:33 DEBUG ----- output ----- 23:02:33 DEBUG { 23:02:33 DEBUG "status": "success", 23:02:33 DEBUG "data": { 23:02:33 DEBUG "resultType": "vector", 23:02:33 DEBUG "result": [ 23:02:33 DEBUG { 23:02:33 DEBUG "metric": { 23:02:33 DEBUG "pod": "ds-idrepo-0" 23:02:33 DEBUG }, 23:02:33 DEBUG "value": [ 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG [...] 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:02:33 DEBUG }, 23:02:33 DEBUG "value": [ 23:02:33 DEBUG 1713391353, 23:02:33 DEBUG "0.000038391798591255264" 23:02:33 DEBUG ] 23:02:33 DEBUG } 23:02:33 DEBUG ] 23:02:33 DEBUG } 23:02:33 DEBUG } 23:02:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391353" 23:02:33 INFO [http_cmd]: http status code OK 23:02:33 DEBUG --- status code --- 23:02:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:33 DEBUG --- http response --- 23:02:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:33 DEBUG ----- output ----- 23:02:33 DEBUG { 23:02:33 DEBUG "status": "success", 23:02:33 DEBUG "data": { 23:02:33 DEBUG "resultType": "vector", 23:02:33 DEBUG "result": [ 23:02:33 DEBUG { 23:02:33 DEBUG "metric": { 23:02:33 DEBUG "pod": "am-b59c765b7-ch5q4" 23:02:33 DEBUG }, 23:02:33 DEBUG "value": [ 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG [...] 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG "pod": "am-b59c765b7-bgsxp" 23:02:33 DEBUG }, 23:02:33 DEBUG "value": [ 23:02:33 DEBUG 1713391353, 23:02:33 DEBUG "1823.3666666666666" 23:02:33 DEBUG ] 23:02:33 DEBUG } 23:02:33 DEBUG ] 23:02:33 DEBUG } 23:02:33 DEBUG } 23:02:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391353" 23:02:33 INFO [http_cmd]: http status code OK 23:02:33 DEBUG --- status code --- 23:02:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:33 DEBUG --- http response --- 23:02:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:33 DEBUG ----- output ----- 23:02:33 DEBUG { 23:02:33 DEBUG "status": "success", 23:02:33 DEBUG "data": { 23:02:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391353" 23:02:33 DEBUG "resultType": "vector", 23:02:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391353" 23:02:33 DEBUG "result": [ 23:02:33 DEBUG { 23:02:33 DEBUG "metric": { 23:02:33 DEBUG "instance": "10.142.0.107:9100" 23:02:33 DEBUG }, 23:02:33 DEBUG "value": [ 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG [...] 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG "instance": "10.142.0.43:9100" 23:02:33 DEBUG }, 23:02:33 DEBUG "value": [ 23:02:33 DEBUG 1713391353, 23:02:33 DEBUG "0.0816333333333328" 23:02:33 DEBUG ] 23:02:33 DEBUG } 23:02:33 DEBUG ] 23:02:33 DEBUG } 23:02:33 DEBUG } 23:02:33 INFO [http_cmd]: http status code OK 23:02:33 DEBUG --- status code --- 23:02:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:33 DEBUG --- http response --- 23:02:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:33 DEBUG ----- output ----- 23:02:33 DEBUG { 23:02:33 DEBUG "status": "success", 23:02:33 DEBUG "data": { 23:02:33 DEBUG "resultType": "vector", 23:02:33 DEBUG "result": [ 23:02:33 DEBUG { 23:02:33 DEBUG "metric": { 23:02:33 DEBUG "instance": "10.142.0.107:9100" 23:02:33 DEBUG }, 23:02:33 DEBUG "value": [ 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG [...] 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG "instance": "10.142.0.43:9100" 23:02:33 DEBUG }, 23:02:33 DEBUG "value": [ 23:02:33 DEBUG 1713391353, 23:02:33 DEBUG "60713460.33333333" 23:02:33 DEBUG ] 23:02:33 DEBUG } 23:02:33 DEBUG ] 23:02:33 DEBUG } 23:02:33 DEBUG } 23:02:33 INFO [http_cmd]: http status code OK 23:02:33 DEBUG --- status code --- 23:02:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:33 DEBUG --- http response --- 23:02:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:33 DEBUG ----- output ----- 23:02:33 DEBUG { 23:02:33 DEBUG "status": "success", 23:02:33 DEBUG "data": { 23:02:33 DEBUG "resultType": "vector", 23:02:33 DEBUG "result": [ 23:02:33 DEBUG { 23:02:33 DEBUG "metric": { 23:02:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:02:33 DEBUG }, 23:02:33 DEBUG "value": [ 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG [...] 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG 23:02:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:02:33 DEBUG }, 23:02:33 DEBUG "value": [ 23:02:33 DEBUG 1713391353, 23:02:33 DEBUG "2262233088" 23:02:33 DEBUG ] 23:02:33 DEBUG } 23:02:33 DEBUG ] 23:02:33 DEBUG } 23:02:33 DEBUG } 23:02:34 INFO 23:02:34 INFO [loop_until]: kubectl --namespace=xlou top pods 23:02:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:02:34 INFO [loop_until]: OK (rc = 0) 23:02:34 DEBUG --- stdout --- 23:02:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11716m 8915Mi am-b59c765b7-bgsxp 11216m 8771Mi am-b59c765b7-ch5q4 11983m 8534Mi ds-cts-0 12318m 7898Mi ds-cts-1 11789m 7767Mi ds-cts-2 12797m 7953Mi ds-idrepo-0 5303m 3220Mi ds-idrepo-1 5409m 2855Mi ds-idrepo-2 5179m 2932Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 3m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6598m 2065Mi 23:02:34 DEBUG --- stderr --- 23:02:34 DEBUG 23:02:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391354" 23:02:34 INFO [http_cmd]: http status code OK 23:02:34 DEBUG --- status code --- 23:02:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:34 DEBUG --- http response --- 23:02:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:02:34 DEBUG ----- output ----- 23:02:34 DEBUG { 23:02:34 DEBUG "status": "success", 23:02:34 DEBUG "data": { 23:02:34 DEBUG "resultType": "vector", 23:02:34 DEBUG "result": [ 23:02:34 DEBUG { 23:02:34 DEBUG "metric": { 23:02:34 DEBUG "pod": "ds-cts-0" 23:02:34 DEBUG }, 23:02:34 DEBUG "value": [ 23:02:34 DEBUG 23:02:34 DEBUG 23:02:34 DEBUG 23:02:34 DEBUG [...] 23:02:34 DEBUG 23:02:34 DEBUG 23:02:34 DEBUG 23:02:34 DEBUG "pod": "ds-cts-2" 23:02:34 DEBUG }, 23:02:34 DEBUG "value": [ 23:02:34 DEBUG 1713391354, 23:02:34 DEBUG "29.866666666666667" 23:02:34 DEBUG ] 23:02:34 DEBUG } 23:02:34 DEBUG ] 23:02:34 DEBUG } 23:02:34 DEBUG } 23:02:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391354" 23:02:34 INFO [http_cmd]: http status code OK 23:02:34 DEBUG --- status code --- 23:02:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:34 DEBUG --- http response --- 23:02:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:34 DEBUG ----- output ----- 23:02:34 DEBUG { 23:02:34 DEBUG "status": "success", 23:02:34 DEBUG "data": { 23:02:34 DEBUG "resultType": "vector", 23:02:34 DEBUG "result": [ 23:02:34 DEBUG { 23:02:34 DEBUG "metric": { 23:02:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:02:34 DEBUG }, 23:02:34 DEBUG "value": [ 23:02:34 DEBUG 23:02:34 DEBUG 23:02:34 DEBUG 23:02:34 DEBUG [...] 23:02:34 DEBUG 23:02:34 DEBUG 23:02:34 DEBUG 23:02:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:02:34 DEBUG }, 23:02:34 DEBUG "value": [ 23:02:34 DEBUG 1713391354, 23:02:34 DEBUG "1805.8999999999999" 23:02:34 DEBUG ] 23:02:34 DEBUG } 23:02:34 DEBUG ] 23:02:34 DEBUG } 23:02:34 DEBUG } 23:02:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391354" 23:02:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391354" 23:02:34 INFO [http_cmd]: http status code OK 23:02:34 INFO [http_cmd]: http status code OK 23:02:34 DEBUG --- status code --- 23:02:34 DEBUG --- status code --- 23:02:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:34 DEBUG --- http response --- 23:02:34 DEBUG --- http response --- 23:02:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:34 DEBUG ----- output ----- 23:02:34 DEBUG { 23:02:34 DEBUG "status": "success", 23:02:34 DEBUG "data": { 23:02:34 DEBUG "resultType": "vector", 23:02:34 DEBUG "result": [ 23:02:34 DEBUG { 23:02:34 DEBUG "metric": { 23:02:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:02:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:02:34 DEBUG }, 23:02:34 DEBUG ----- output ----- 23:02:34 DEBUG "value": [ 23:02:34 DEBUG { 23:02:34 DEBUG 23:02:34 DEBUG "status": "success", 23:02:34 DEBUG 23:02:34 DEBUG "data": { 23:02:34 DEBUG 23:02:34 DEBUG "resultType": "vector", 23:02:34 DEBUG [...] 23:02:34 DEBUG "result": [ 23:02:34 DEBUG 23:02:34 DEBUG { 23:02:34 DEBUG 23:02:34 DEBUG "metric": { 23:02:34 DEBUG 23:02:34 DEBUG "pod": "ds-idrepo-0" 23:02:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:02:34 DEBUG }, 23:02:34 DEBUG }, 23:02:34 DEBUG "value": [ 23:02:34 DEBUG "value": [ 23:02:34 DEBUG 23:02:34 DEBUG 1713391354, 23:02:34 DEBUG 23:02:34 DEBUG "1823.5" 23:02:34 DEBUG 23:02:34 DEBUG ] 23:02:34 DEBUG [...] 23:02:34 DEBUG } 23:02:34 DEBUG 23:02:34 DEBUG ] 23:02:34 DEBUG 23:02:34 DEBUG } 23:02:34 DEBUG 23:02:34 DEBUG } 23:02:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:02:34 DEBUG }, 23:02:34 DEBUG "value": [ 23:02:34 DEBUG 1713391354, 23:02:34 DEBUG "0" 23:02:34 DEBUG ] 23:02:34 DEBUG } 23:02:34 DEBUG ] 23:02:34 DEBUG } 23:02:34 DEBUG } 23:02:34 INFO 23:02:34 INFO [loop_until]: kubectl --namespace=xlou top node 23:02:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:02:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391354" 23:02:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391354" 23:02:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391354" 23:02:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391354" 23:02:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391354" 23:02:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391354" 23:02:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391354" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391355" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391355" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391355" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391355" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391355" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391355" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391355" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391355" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391355" 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG --- status code --- 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG "data": { 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG { 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG { 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:02:35 DEBUG { 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "result": [ 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG "data": { 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG { 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "data": { 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG { 23:02:35 DEBUG "result": [ 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG { 23:02:35 DEBUG "data": { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "result": [ 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "result": [ 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG "data": { 23:02:35 DEBUG "data": { 23:02:35 DEBUG "data": { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "data": { 23:02:35 DEBUG { 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "pod": "ds-cts-0" 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "result": [ 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:02:35 DEBUG "pod": "ds-idrepo-0" 23:02:35 DEBUG }, 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "data": { 23:02:35 DEBUG { 23:02:35 DEBUG { 23:02:35 DEBUG "result": [ 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG }, 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG "pod": "ds-cts-0" 23:02:35 DEBUG }, 23:02:35 DEBUG "result": [ 23:02:35 DEBUG 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "ds-idrepo-0" 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "ds-idrepo-0" 23:02:35 DEBUG "value": [ 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG { 23:02:35 DEBUG "result": [ 23:02:35 DEBUG "value": [ 23:02:35 DEBUG { 23:02:35 DEBUG 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "instance": "10.142.0.107:9100" 23:02:35 DEBUG }, 23:02:35 DEBUG }, 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "instance": "10.142.0.107:9100" 23:02:35 DEBUG "pod": "ds-idrepo-0" 23:02:35 DEBUG }, 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG { 23:02:35 DEBUG 23:02:35 DEBUG "metric": { 23:02:35 DEBUG 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG "result": [ 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG "data": { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "instance": "10.142.0.107:9100" 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:02:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG 23:02:35 DEBUG "pod": "ds-idrepo-0" 23:02:35 DEBUG }, 23:02:35 DEBUG 23:02:35 DEBUG "value": [ 23:02:35 DEBUG }, 23:02:35 DEBUG [...] 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG "result": [ 23:02:35 INFO [loop_until]: OK (rc = 0) 23:02:35 DEBUG }, 23:02:35 DEBUG }, 23:02:35 DEBUG "pod": "ds-cts-2" 23:02:35 DEBUG [...] 23:02:35 DEBUG "pod": "ds-idrepo-0" 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG { 23:02:35 DEBUG --- stdout --- 23:02:35 DEBUG "value": [ 23:02:35 DEBUG "value": [ 23:02:35 DEBUG }, 23:02:35 DEBUG 23:02:35 DEBUG }, 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "ds-cts-2" 23:02:35 DEBUG "metric": { 23:02:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5452Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5256m 17% 6143Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11483m 38% 11121Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 2994Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12501m 41% 10132Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 166m 0% 4966Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11867m 39% 11515Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27872m 93% 2923Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11881m 39% 10604Mi 9% gke-xlou-cdm-ds-0f601751-j488 5517m 18% 5503Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11625m 38% 11300Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5795m 19% 5978Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12369m 41% 9669Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6526m 41% 5845Mi 9% 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG "value": [ 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "instance": "10.142.0.43:9100" 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG }, 23:02:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:02:35 DEBUG --- stderr --- 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 1713391355, 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG "pod": "ds-cts-2" 23:02:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:02:35 DEBUG "value": [ 23:02:35 DEBUG }, 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG "0.000046431111111111106" 23:02:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG }, 23:02:35 DEBUG [...] 23:02:35 DEBUG }, 23:02:35 DEBUG }, 23:02:35 DEBUG 1713391355, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG ] 23:02:35 DEBUG }, 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "ds-idrepo-2" 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG "value": [ 23:02:35 DEBUG "value": [ 23:02:35 DEBUG "0" 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG } 23:02:35 DEBUG "value": [ 23:02:35 DEBUG [...] 23:02:35 DEBUG "instance": "10.142.0.43:9100" 23:02:35 DEBUG }, 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 1713391354, 23:02:35 DEBUG 23:02:35 DEBUG 1713391354, 23:02:35 DEBUG 1713391354, 23:02:35 DEBUG ] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG ] 23:02:35 DEBUG 1713391354, 23:02:35 DEBUG "6.587231627941748" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:02:35 DEBUG }, 23:02:35 DEBUG "207605357568" 23:02:35 DEBUG 23:02:35 DEBUG "0" 23:02:35 DEBUG "0.018" 23:02:35 DEBUG } 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:02:35 DEBUG } 23:02:35 DEBUG 23:02:35 DEBUG ] 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713391355, 23:02:35 DEBUG 23:02:35 DEBUG "pod": "ds-idrepo-2" 23:02:35 DEBUG "value": [ 23:02:35 DEBUG ] 23:02:35 DEBUG "pod": "ds-idrepo-2" 23:02:35 DEBUG ] 23:02:35 DEBUG ] 23:02:35 DEBUG ] 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG }, 23:02:35 DEBUG } 23:02:35 DEBUG 23:02:35 DEBUG } 23:02:35 DEBUG 1713391355, 23:02:35 DEBUG "0" 23:02:35 DEBUG "instance": "10.142.0.43:9100" 23:02:35 DEBUG }, 23:02:35 DEBUG 1713391354, 23:02:35 DEBUG } 23:02:35 DEBUG }, 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG ] 23:02:35 DEBUG "0.0536666666666668" 23:02:35 DEBUG ] 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713391355, 23:02:35 DEBUG ] 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713391355, 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:02:35 DEBUG 23:02:35 DEBUG 1713391355, 23:02:35 DEBUG "pod": "ds-idrepo-2" 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG "60712781.5" 23:02:35 DEBUG "value": [ 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG "0" 23:02:35 DEBUG ] 23:02:35 DEBUG "0" 23:02:35 DEBUG ] 23:02:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:02:35 DEBUG "0" 23:02:35 DEBUG }, 23:02:35 DEBUG ] 23:02:35 DEBUG ] 23:02:35 DEBUG ] 23:02:35 DEBUG 1713391355, 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG }, 23:02:35 DEBUG ] 23:02:35 DEBUG "value": [ 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG "0" 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG "value": [ 23:02:35 DEBUG } 23:02:35 DEBUG 1713391355, 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG 1713391354, 23:02:35 DEBUG 1713391354, 23:02:35 DEBUG "328956" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG "1823.2666666666667" 23:02:35 DEBUG ] 23:02:35 DEBUG "0" 23:02:35 DEBUG ] 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:03:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391413" 23:03:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391413" 23:03:33 ERROR [http_cmd]: ERROR 23:03:33 ERROR --- status code --- 23:03:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:03:33 ERROR --- http response --- 23:03:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:03:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:03:33 INFO [http_cmd]: http status code OK 23:03:33 DEBUG --- status code --- 23:03:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:33 DEBUG --- http response --- 23:03:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:03:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391413" 23:03:33 INFO [http_cmd]: http status code OK 23:03:33 DEBUG --- status code --- 23:03:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:33 DEBUG --- http response --- 23:03:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:03:33 DEBUG ----- output ----- 23:03:33 DEBUG { 23:03:33 DEBUG "status": "success", 23:03:33 DEBUG "data": { 23:03:33 DEBUG "resultType": "vector", 23:03:33 DEBUG "result": [ 23:03:33 DEBUG { 23:03:33 DEBUG "metric": { 23:03:33 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:03:33 DEBUG }, 23:03:33 DEBUG "value": [ 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG [...] 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:03:33 DEBUG }, 23:03:33 DEBUG "value": [ 23:03:33 DEBUG 1713391413, 23:03:33 DEBUG "0" 23:03:33 DEBUG ] 23:03:33 DEBUG } 23:03:33 DEBUG ] 23:03:33 DEBUG } 23:03:33 DEBUG } 23:03:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391413" 23:03:33 INFO [http_cmd]: http status code OK 23:03:33 DEBUG --- status code --- 23:03:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:33 DEBUG --- http response --- 23:03:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:33 DEBUG ----- output ----- 23:03:33 DEBUG { 23:03:33 DEBUG "status": "success", 23:03:33 DEBUG "data": { 23:03:33 DEBUG "resultType": "vector", 23:03:33 DEBUG "result": [ 23:03:33 DEBUG { 23:03:33 DEBUG "metric": { 23:03:33 DEBUG "pod": "am-b59c765b7-ch5q4" 23:03:33 DEBUG }, 23:03:33 DEBUG "value": [ 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG [...] 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG "pod": "am-b59c765b7-bgsxp" 23:03:33 DEBUG }, 23:03:33 DEBUG "value": [ 23:03:33 DEBUG 1713391413, 23:03:33 DEBUG "1830.7666666666667" 23:03:33 DEBUG ] 23:03:33 DEBUG } 23:03:33 DEBUG ] 23:03:33 DEBUG } 23:03:33 DEBUG } 23:03:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391413" 23:03:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391413" 23:03:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391413" 23:03:33 INFO [http_cmd]: http status code OK 23:03:33 DEBUG --- status code --- 23:03:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:33 DEBUG --- http response --- 23:03:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:33 DEBUG ----- output ----- 23:03:33 DEBUG { 23:03:33 DEBUG "status": "success", 23:03:33 DEBUG "data": { 23:03:33 DEBUG "resultType": "vector", 23:03:33 DEBUG "result": [ 23:03:33 DEBUG { 23:03:33 DEBUG "metric": { 23:03:33 DEBUG "instance": "10.142.0.107:9100" 23:03:33 DEBUG }, 23:03:33 DEBUG "value": [ 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG [...] 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG "instance": "10.142.0.43:9100" 23:03:33 DEBUG }, 23:03:33 DEBUG "value": [ 23:03:33 DEBUG 1713391413, 23:03:33 DEBUG "0.06370000000000003" 23:03:33 DEBUG ] 23:03:33 DEBUG } 23:03:33 DEBUG ] 23:03:33 DEBUG } 23:03:33 DEBUG } 23:03:33 INFO [http_cmd]: http status code OK 23:03:33 DEBUG --- status code --- 23:03:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:33 DEBUG --- http response --- 23:03:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:33 DEBUG ----- output ----- 23:03:33 DEBUG { 23:03:33 DEBUG "status": "success", 23:03:33 DEBUG "data": { 23:03:33 DEBUG "resultType": "vector", 23:03:33 DEBUG "result": [ 23:03:33 DEBUG { 23:03:33 DEBUG "metric": { 23:03:33 DEBUG "instance": "10.142.0.107:9100" 23:03:33 DEBUG }, 23:03:33 DEBUG "value": [ 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG [...] 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG "instance": "10.142.0.43:9100" 23:03:33 DEBUG }, 23:03:33 DEBUG "value": [ 23:03:33 DEBUG 1713391413, 23:03:33 DEBUG "61024386.93333333" 23:03:33 DEBUG ] 23:03:33 DEBUG } 23:03:33 DEBUG ] 23:03:33 DEBUG } 23:03:33 DEBUG } 23:03:33 INFO [http_cmd]: http status code OK 23:03:33 DEBUG --- status code --- 23:03:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:33 DEBUG --- http response --- 23:03:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:33 DEBUG ----- output ----- 23:03:33 DEBUG { 23:03:33 DEBUG "status": "success", 23:03:33 DEBUG "data": { 23:03:33 DEBUG "resultType": "vector", 23:03:33 DEBUG "result": [ 23:03:33 DEBUG { 23:03:33 DEBUG "metric": { 23:03:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:03:33 DEBUG }, 23:03:33 DEBUG "value": [ 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG [...] 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG 23:03:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:03:33 DEBUG }, 23:03:33 DEBUG "value": [ 23:03:33 DEBUG 1713391413, 23:03:33 DEBUG "2261835776" 23:03:33 DEBUG ] 23:03:33 DEBUG } 23:03:33 DEBUG ] 23:03:33 DEBUG } 23:03:33 DEBUG } 23:03:34 INFO 23:03:34 INFO [loop_until]: kubectl --namespace=xlou top pods 23:03:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:03:34 INFO [loop_until]: OK (rc = 0) 23:03:34 DEBUG --- stdout --- 23:03:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12057m 8917Mi am-b59c765b7-bgsxp 11331m 8770Mi am-b59c765b7-ch5q4 12028m 8535Mi ds-cts-0 12475m 7884Mi ds-cts-1 11440m 7747Mi ds-cts-2 13060m 7961Mi ds-idrepo-0 5427m 3220Mi ds-idrepo-1 5435m 2855Mi ds-idrepo-2 5181m 2932Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 967Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6628m 2066Mi 23:03:34 DEBUG --- stderr --- 23:03:34 DEBUG 23:03:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391414" 23:03:34 INFO [http_cmd]: http status code OK 23:03:34 DEBUG --- status code --- 23:03:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:34 DEBUG --- http response --- 23:03:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:03:34 DEBUG ----- output ----- 23:03:34 DEBUG { 23:03:34 DEBUG "status": "success", 23:03:34 DEBUG "data": { 23:03:34 DEBUG "resultType": "vector", 23:03:34 DEBUG "result": [ 23:03:34 DEBUG { 23:03:34 DEBUG "metric": { 23:03:34 DEBUG "pod": "ds-cts-0" 23:03:34 DEBUG }, 23:03:34 DEBUG "value": [ 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG [...] 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG "pod": "ds-cts-2" 23:03:34 DEBUG }, 23:03:34 DEBUG "value": [ 23:03:34 DEBUG 1713391414, 23:03:34 DEBUG "29.866666666666667" 23:03:34 DEBUG ] 23:03:34 DEBUG } 23:03:34 DEBUG ] 23:03:34 DEBUG } 23:03:34 DEBUG } 23:03:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391414" 23:03:34 INFO [http_cmd]: http status code OK 23:03:34 DEBUG --- status code --- 23:03:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:34 DEBUG --- http response --- 23:03:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:34 DEBUG ----- output ----- 23:03:34 DEBUG { 23:03:34 DEBUG "status": "success", 23:03:34 DEBUG "data": { 23:03:34 DEBUG "resultType": "vector", 23:03:34 DEBUG "result": [ 23:03:34 DEBUG { 23:03:34 DEBUG "metric": { 23:03:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:03:34 DEBUG }, 23:03:34 DEBUG "value": [ 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG [...] 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:03:34 DEBUG }, 23:03:34 DEBUG "value": [ 23:03:34 DEBUG 1713391414, 23:03:34 DEBUG "1824.1666666666667" 23:03:34 DEBUG ] 23:03:34 DEBUG } 23:03:34 DEBUG ] 23:03:34 DEBUG } 23:03:34 DEBUG } 23:03:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391414" 23:03:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391414" 23:03:34 INFO [http_cmd]: http status code OK 23:03:34 INFO [http_cmd]: http status code OK 23:03:34 DEBUG --- status code --- 23:03:34 DEBUG --- status code --- 23:03:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:34 DEBUG --- http response --- 23:03:34 DEBUG --- http response --- 23:03:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:03:34 DEBUG ----- output ----- 23:03:34 DEBUG ----- output ----- 23:03:34 DEBUG { 23:03:34 DEBUG { 23:03:34 DEBUG "status": "success", 23:03:34 DEBUG "status": "success", 23:03:34 DEBUG "data": { 23:03:34 DEBUG "data": { 23:03:34 DEBUG "resultType": "vector", 23:03:34 DEBUG "resultType": "vector", 23:03:34 DEBUG "result": [ 23:03:34 DEBUG "result": [ 23:03:34 DEBUG { 23:03:34 DEBUG { 23:03:34 DEBUG "metric": { 23:03:34 DEBUG "metric": { 23:03:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:03:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:03:34 DEBUG }, 23:03:34 DEBUG }, 23:03:34 DEBUG "value": [ 23:03:34 DEBUG "value": [ 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG [...] 23:03:34 DEBUG [...] 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG 23:03:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:03:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:03:34 DEBUG }, 23:03:34 DEBUG }, 23:03:34 DEBUG "value": [ 23:03:34 DEBUG "value": [ 23:03:34 DEBUG 1713391414, 23:03:34 DEBUG 1713391414, 23:03:34 DEBUG "1830.5666666666666" 23:03:34 DEBUG "0" 23:03:34 DEBUG ] 23:03:34 DEBUG ] 23:03:34 DEBUG } 23:03:34 DEBUG } 23:03:34 DEBUG ] 23:03:34 DEBUG ] 23:03:34 DEBUG } 23:03:34 DEBUG } 23:03:34 DEBUG } 23:03:34 DEBUG } 23:03:35 INFO 23:03:35 INFO [loop_until]: kubectl --namespace=xlou top node 23:03:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391415" 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391415" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391415" 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG { 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG { 23:03:35 DEBUG { 23:03:35 DEBUG { 23:03:35 DEBUG { 23:03:35 DEBUG { 23:03:35 DEBUG { 23:03:35 DEBUG { 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG { 23:03:35 DEBUG "data": { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "data": { 23:03:35 DEBUG { 23:03:35 DEBUG "data": { 23:03:35 DEBUG "data": { 23:03:35 DEBUG "data": { 23:03:35 DEBUG "data": { 23:03:35 DEBUG "data": { 23:03:35 DEBUG "data": { 23:03:35 DEBUG "data": { 23:03:35 DEBUG { 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "result": [ 23:03:35 DEBUG "data": { 23:03:35 DEBUG "result": [ 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG { 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "result": [ 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG "result": [ 23:03:35 DEBUG "data": { 23:03:35 DEBUG "data": { 23:03:35 DEBUG "result": [ 23:03:35 DEBUG "result": [ 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "data": { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG { 23:03:35 DEBUG "pod": "ds-idrepo-0" 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG { 23:03:35 DEBUG { 23:03:35 DEBUG "instance": "10.142.0.107:9100" 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "pod": "ds-idrepo-0" 23:03:35 DEBUG "metric": { 23:03:35 DEBUG }, 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "result": [ 23:03:35 DEBUG "result": [ 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:03:35 DEBUG "result": [ 23:03:35 DEBUG }, 23:03:35 DEBUG "pod": "ds-cts-0" 23:03:35 DEBUG "value": [ 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "ds-idrepo-0" 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:03:35 DEBUG "instance": "10.142.0.107:9100" 23:03:35 DEBUG { 23:03:35 DEBUG { 23:03:35 DEBUG "pod": "ds-cts-0" 23:03:35 DEBUG "pod": "ds-idrepo-0" 23:03:35 DEBUG 23:03:35 DEBUG }, 23:03:35 DEBUG "pod": "ds-idrepo-0" 23:03:35 DEBUG { 23:03:35 DEBUG "value": [ 23:03:35 DEBUG }, 23:03:35 DEBUG 23:03:35 DEBUG "instance": "10.142.0.107:9100" 23:03:35 DEBUG }, 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:03:35 DEBUG }, 23:03:35 DEBUG 23:03:35 DEBUG "value": [ 23:03:35 DEBUG }, 23:03:35 DEBUG "metric": { 23:03:35 DEBUG 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:03:35 DEBUG "metric": { 23:03:35 DEBUG 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG }, 23:03:35 DEBUG "pod": "ds-idrepo-0" 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG }, 23:03:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG "value": [ 23:03:35 DEBUG }, 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "value": [ 23:03:35 DEBUG }, 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "value": [ 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG "value": [ 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "ds-cts-2" 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "instance": "10.142.0.43:9100" 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG "pod": "idm-b85f9d978-txpwv" 23:03:35 DEBUG 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG "instance": "10.142.0.43:9100" 23:03:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:03:35 DEBUG }, 23:03:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG }, 23:03:35 DEBUG "pod": "ds-cts-2" 23:03:35 INFO [loop_until]: OK (rc = 0) 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG 23:03:35 DEBUG "pod": "ds-idrepo-2" 23:03:35 DEBUG }, 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG }, 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG }, 23:03:35 DEBUG 23:03:35 DEBUG "pod": "ds-idrepo-2" 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "value": [ 23:03:35 DEBUG }, 23:03:35 DEBUG --- stdout --- 23:03:35 DEBUG "0" 23:03:35 DEBUG "instance": "10.142.0.43:9100" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "value": [ 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG "pod": "ds-cts-2" 23:03:35 DEBUG }, 23:03:35 DEBUG "pod": "idm-b85f9d978-txpwv" 23:03:35 DEBUG 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG ] 23:03:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5440Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5293m 17% 6151Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11870m 39% 11124Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 77m 0% 2994Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12663m 42% 10141Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 4969Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11930m 39% 11521Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27885m 93% 2924Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11854m 39% 10619Mi 9% gke-xlou-cdm-ds-0f601751-j488 5518m 18% 5517Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11673m 39% 11314Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5800m 19% 5996Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12679m 42% 9660Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6724m 42% 5844Mi 9% 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:03:35 DEBUG 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG "210712710144" 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG }, 23:03:35 DEBUG "pod": "ds-idrepo-2" 23:03:35 DEBUG "0" 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG } 23:03:35 DEBUG --- stderr --- 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG "0.018" 23:03:35 DEBUG }, 23:03:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:03:35 DEBUG "0.043666666666666167" 23:03:35 DEBUG "329366" 23:03:35 DEBUG ] 23:03:35 DEBUG ] 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG }, 23:03:35 DEBUG ] 23:03:35 DEBUG "0" 23:03:35 DEBUG ] 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG 23:03:35 DEBUG "0" 23:03:35 DEBUG ] 23:03:35 DEBUG "value": [ 23:03:35 DEBUG }, 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG "6.692147386978628" 23:03:35 DEBUG } 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG "0" 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG "61023779.83333333" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG ] 23:03:35 DEBUG ] 23:03:35 DEBUG "0.000033333333333333335" 23:03:35 DEBUG ] 23:03:35 DEBUG "217.44244605804838" 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG "0" 23:03:35 DEBUG 1713391415, 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG "0" 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG "1830.9333333333334" 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:04:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391473" 23:04:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391473" 23:04:33 ERROR [http_cmd]: ERROR 23:04:33 INFO [http_cmd]: http status code OK 23:04:33 ERROR --- status code --- 23:04:33 DEBUG --- status code --- 23:04:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:04:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:33 ERROR --- http response --- 23:04:33 DEBUG --- http response --- 23:04:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:04:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:04:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:04:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391473" 23:04:33 INFO [http_cmd]: http status code OK 23:04:33 DEBUG --- status code --- 23:04:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:33 DEBUG --- http response --- 23:04:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:33 DEBUG ----- output ----- 23:04:33 DEBUG { 23:04:33 DEBUG "status": "success", 23:04:33 DEBUG "data": { 23:04:33 DEBUG "resultType": "vector", 23:04:33 DEBUG "result": [ 23:04:33 DEBUG { 23:04:33 DEBUG "metric": { 23:04:33 DEBUG "pod": "ds-idrepo-1" 23:04:33 DEBUG }, 23:04:33 DEBUG "value": [ 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG [...] 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 23:04:33 DEBUG }, 23:04:33 DEBUG "value": [ 23:04:33 DEBUG 1713391473, 23:04:33 DEBUG "0" 23:04:33 DEBUG ] 23:04:33 DEBUG } 23:04:33 DEBUG ] 23:04:33 DEBUG } 23:04:33 DEBUG } 23:04:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391473" 23:04:33 INFO [http_cmd]: http status code OK 23:04:33 DEBUG --- status code --- 23:04:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:33 DEBUG --- http response --- 23:04:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:33 DEBUG ----- output ----- 23:04:33 DEBUG { 23:04:33 DEBUG "status": "success", 23:04:33 DEBUG "data": { 23:04:33 DEBUG "resultType": "vector", 23:04:33 DEBUG "result": [ 23:04:33 DEBUG { 23:04:33 DEBUG "metric": { 23:04:33 DEBUG "pod": "am-b59c765b7-ch5q4" 23:04:33 DEBUG }, 23:04:33 DEBUG "value": [ 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG [...] 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG "pod": "am-b59c765b7-bgsxp" 23:04:33 DEBUG }, 23:04:33 DEBUG "value": [ 23:04:33 DEBUG 1713391473, 23:04:33 DEBUG "1819.8666666666666" 23:04:33 DEBUG ] 23:04:33 DEBUG } 23:04:33 DEBUG ] 23:04:33 DEBUG } 23:04:33 DEBUG } 23:04:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391473" 23:04:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391473" 23:04:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391473" 23:04:33 INFO [http_cmd]: http status code OK 23:04:33 DEBUG --- status code --- 23:04:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:33 DEBUG --- http response --- 23:04:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:33 DEBUG ----- output ----- 23:04:33 DEBUG { 23:04:33 DEBUG "status": "success", 23:04:33 DEBUG "data": { 23:04:33 DEBUG "resultType": "vector", 23:04:33 DEBUG "result": [ 23:04:33 DEBUG { 23:04:33 DEBUG "metric": { 23:04:33 DEBUG "instance": "10.142.0.107:9100" 23:04:33 DEBUG }, 23:04:33 DEBUG "value": [ 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG [...] 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG "instance": "10.142.0.43:9100" 23:04:33 DEBUG }, 23:04:33 DEBUG "value": [ 23:04:33 DEBUG 1713391473, 23:04:33 DEBUG "0.15256666666666754" 23:04:33 DEBUG ] 23:04:33 DEBUG } 23:04:33 DEBUG ] 23:04:33 DEBUG } 23:04:33 DEBUG } 23:04:33 INFO [http_cmd]: http status code OK 23:04:33 DEBUG --- status code --- 23:04:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:33 DEBUG --- http response --- 23:04:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:33 DEBUG ----- output ----- 23:04:33 DEBUG { 23:04:33 DEBUG "status": "success", 23:04:33 DEBUG "data": { 23:04:33 DEBUG "resultType": "vector", 23:04:33 DEBUG "result": [ 23:04:33 DEBUG { 23:04:33 DEBUG "metric": { 23:04:33 DEBUG "instance": "10.142.0.107:9100" 23:04:33 DEBUG }, 23:04:33 DEBUG "value": [ 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG [...] 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 INFO [http_cmd]: http status code OK 23:04:33 DEBUG "instance": "10.142.0.43:9100" 23:04:33 DEBUG --- status code --- 23:04:33 DEBUG }, 23:04:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:33 DEBUG "value": [ 23:04:33 DEBUG --- http response --- 23:04:33 DEBUG 1713391473, 23:04:33 DEBUG "61118787.5" 23:04:33 DEBUG ] 23:04:33 DEBUG } 23:04:33 DEBUG ] 23:04:33 DEBUG } 23:04:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:33 DEBUG } 23:04:33 DEBUG ----- output ----- 23:04:33 DEBUG { 23:04:33 DEBUG "status": "success", 23:04:33 DEBUG "data": { 23:04:33 DEBUG "resultType": "vector", 23:04:33 DEBUG "result": [ 23:04:33 DEBUG { 23:04:33 DEBUG "metric": { 23:04:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:04:33 DEBUG }, 23:04:33 DEBUG "value": [ 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG [...] 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG 23:04:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:04:33 DEBUG }, 23:04:33 DEBUG "value": [ 23:04:33 DEBUG 1713391473, 23:04:33 DEBUG "2265133056" 23:04:33 DEBUG ] 23:04:33 DEBUG } 23:04:33 DEBUG ] 23:04:33 DEBUG } 23:04:33 DEBUG } 23:04:34 INFO 23:04:34 INFO [loop_until]: kubectl --namespace=xlou top pods 23:04:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:04:34 INFO [loop_until]: OK (rc = 0) 23:04:34 DEBUG --- stdout --- 23:04:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11785m 8915Mi am-b59c765b7-bgsxp 11260m 8770Mi am-b59c765b7-ch5q4 11771m 8535Mi ds-cts-0 12215m 7891Mi ds-cts-1 11625m 7764Mi ds-cts-2 13180m 7961Mi ds-idrepo-0 5375m 3220Mi ds-idrepo-1 5221m 2879Mi ds-idrepo-2 5109m 2933Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 3m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6774m 2094Mi 23:04:34 DEBUG --- stderr --- 23:04:34 DEBUG 23:04:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391474" 23:04:34 INFO [http_cmd]: http status code OK 23:04:34 DEBUG --- status code --- 23:04:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:34 DEBUG --- http response --- 23:04:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:04:34 DEBUG ----- output ----- 23:04:34 DEBUG { 23:04:34 DEBUG "status": "success", 23:04:34 DEBUG "data": { 23:04:34 DEBUG "resultType": "vector", 23:04:34 DEBUG "result": [ 23:04:34 DEBUG { 23:04:34 DEBUG "metric": { 23:04:34 DEBUG "pod": "ds-cts-0" 23:04:34 DEBUG }, 23:04:34 DEBUG "value": [ 23:04:34 DEBUG 23:04:34 DEBUG 23:04:34 DEBUG 23:04:34 DEBUG [...] 23:04:34 DEBUG 23:04:34 DEBUG 23:04:34 DEBUG 23:04:34 DEBUG "pod": "ds-cts-2" 23:04:34 DEBUG }, 23:04:34 DEBUG "value": [ 23:04:34 DEBUG 1713391474, 23:04:34 DEBUG "29.966666666666665" 23:04:34 DEBUG ] 23:04:34 DEBUG } 23:04:34 DEBUG ] 23:04:34 DEBUG } 23:04:34 DEBUG } 23:04:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391474" 23:04:34 INFO [http_cmd]: http status code OK 23:04:34 DEBUG --- status code --- 23:04:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:34 DEBUG --- http response --- 23:04:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:34 DEBUG ----- output ----- 23:04:34 DEBUG { 23:04:34 DEBUG "status": "success", 23:04:34 DEBUG "data": { 23:04:34 DEBUG "resultType": "vector", 23:04:34 DEBUG "result": [ 23:04:34 DEBUG { 23:04:34 DEBUG "metric": { 23:04:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:04:34 DEBUG }, 23:04:34 DEBUG "value": [ 23:04:34 DEBUG 23:04:34 DEBUG 23:04:34 DEBUG 23:04:34 DEBUG [...] 23:04:34 DEBUG 23:04:34 DEBUG 23:04:34 DEBUG 23:04:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:04:34 DEBUG }, 23:04:34 DEBUG "value": [ 23:04:34 DEBUG 1713391474, 23:04:34 DEBUG "1826.9666666666667" 23:04:34 DEBUG ] 23:04:34 DEBUG } 23:04:34 DEBUG ] 23:04:34 DEBUG } 23:04:34 DEBUG } 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:04:35 DEBUG "pod": "ds-idrepo-0" 23:04:35 DEBUG }, 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "pod": "idm-b85f9d978-txpwv" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG "0" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG "1820.1333333333334" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 INFO 23:04:35 INFO [loop_until]: kubectl --namespace=xlou top node 23:04:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391475" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391475" 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 INFO [loop_until]: OK (rc = 0) 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG --- status code --- 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG --- stdout --- 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 5436Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5273m 17% 6153Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11890m 39% 11113Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 2991Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12422m 41% 10143Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 4970Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12031m 40% 11520Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27181m 91% 2922Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11776m 39% 10608Mi 9% gke-xlou-cdm-ds-0f601751-j488 5504m 18% 5516Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11423m 38% 11317Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5816m 19% 5998Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12993m 43% 9663Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6832m 42% 5867Mi 10% 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG --- stderr --- 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:04:35 DEBUG --- http response --- 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG { 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG { 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "data": { 23:04:35 DEBUG "data": { 23:04:35 DEBUG "data": { 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG "data": { 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "data": { 23:04:35 DEBUG { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG "result": [ 23:04:35 DEBUG "result": [ 23:04:35 DEBUG "result": [ 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG "result": [ 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "data": { 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG "result": [ 23:04:35 DEBUG "pod": "ds-idrepo-0" 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "ds-idrepo-0" 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "instance": "10.142.0.107:9100" 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG { 23:04:35 DEBUG }, 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "instance": "10.142.0.107:9100" 23:04:35 DEBUG "pod": "ds-cts-0" 23:04:35 DEBUG "pod": "ds-idrepo-0" 23:04:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:04:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:04:35 DEBUG }, 23:04:35 DEBUG "pod": "ds-idrepo-0" 23:04:35 DEBUG }, 23:04:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:04:35 DEBUG }, 23:04:35 DEBUG "result": [ 23:04:35 DEBUG "pod": "ds-idrepo-0" 23:04:35 DEBUG "pod": "ds-cts-0" 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "value": [ 23:04:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:04:35 DEBUG }, 23:04:35 DEBUG }, 23:04:35 DEBUG }, 23:04:35 DEBUG }, 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG { 23:04:35 DEBUG }, 23:04:35 DEBUG }, 23:04:35 DEBUG "pod": "ds-idrepo-0" 23:04:35 DEBUG 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG "value": [ 23:04:35 DEBUG "value": [ 23:04:35 DEBUG "value": [ 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "instance": "10.142.0.107:9100" 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG }, 23:04:35 DEBUG 23:04:35 DEBUG }, 23:04:35 DEBUG [...] 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG [...] 23:04:35 DEBUG [...] 23:04:35 DEBUG [...] 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "pod": "ds-idrepo-2" 23:04:35 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "pod": "ds-idrepo-2" 23:04:35 DEBUG 23:04:35 DEBUG }, 23:04:35 DEBUG 23:04:35 DEBUG "pod": "ds-cts-2" 23:04:35 DEBUG 23:04:35 DEBUG "instance": "10.142.0.43:9100" 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG }, 23:04:35 DEBUG }, 23:04:35 DEBUG 23:04:35 DEBUG "instance": "10.142.0.43:9100" 23:04:35 DEBUG "pod": "ds-cts-2" 23:04:35 DEBUG }, 23:04:35 DEBUG 23:04:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG }, 23:04:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:04:35 DEBUG }, 23:04:35 DEBUG 23:04:35 DEBUG "pod": "ds-cts-2" 23:04:35 DEBUG 23:04:35 DEBUG "value": [ 23:04:35 DEBUG "value": [ 23:04:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:04:35 DEBUG }, 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG "value": [ 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG }, 23:04:35 DEBUG 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG }, 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG "6.618745268525772" 23:04:35 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG "0" 23:04:35 DEBUG "0" 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG "0" 23:04:35 DEBUG "value": [ 23:04:35 DEBUG "329516.5" 23:04:35 DEBUG ] 23:04:35 DEBUG }, 23:04:35 DEBUG "0" 23:04:35 DEBUG "1819.5333333333333" 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG "instance": "10.142.0.43:9100" 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG "pod": "ds-idrepo-2" 23:04:35 DEBUG ] 23:04:35 DEBUG ] 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG "0.07766666666666672" 23:04:35 DEBUG "0" 23:04:35 DEBUG ] 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG "value": [ 23:04:35 DEBUG ] 23:04:35 DEBUG ] 23:04:35 DEBUG "213711893504" 23:04:35 DEBUG }, 23:04:35 DEBUG "0.0008666666666666666" 23:04:35 DEBUG }, 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG "0" 23:04:35 DEBUG ] 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG "0.018" 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG "value": [ 23:04:35 DEBUG ] 23:04:35 DEBUG "value": [ 23:04:35 DEBUG ] 23:04:35 DEBUG ] 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG ] 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG "0" 23:04:35 DEBUG ] 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG 1713391475, 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG "61122370.43333333" 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG "0" 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG ] 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 DEBUG } 23:05:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391533" 23:05:33 INFO [http_cmd]: http status code OK 23:05:33 DEBUG --- status code --- 23:05:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:33 DEBUG --- http response --- 23:05:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:05:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391533" 23:05:33 ERROR [http_cmd]: ERROR 23:05:33 ERROR --- status code --- 23:05:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:05:33 ERROR --- http response --- 23:05:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:05:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:05:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391533" 23:05:33 INFO [http_cmd]: http status code OK 23:05:33 DEBUG --- status code --- 23:05:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:33 DEBUG --- http response --- 23:05:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:05:33 DEBUG ----- output ----- 23:05:33 DEBUG { 23:05:33 DEBUG "status": "success", 23:05:33 DEBUG "data": { 23:05:33 DEBUG "resultType": "vector", 23:05:33 DEBUG "result": [ 23:05:33 DEBUG { 23:05:33 DEBUG "metric": { 23:05:33 DEBUG "pod": "ds-idrepo-0" 23:05:33 DEBUG }, 23:05:33 DEBUG "value": [ 23:05:33 DEBUG 23:05:33 DEBUG 23:05:33 DEBUG 23:05:33 DEBUG [...] 23:05:33 DEBUG 23:05:33 DEBUG 23:05:33 DEBUG 23:05:33 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:05:33 DEBUG }, 23:05:33 DEBUG "value": [ 23:05:33 DEBUG 1713391533, 23:05:33 DEBUG "0.0030299669244733224" 23:05:33 DEBUG ] 23:05:33 DEBUG } 23:05:33 DEBUG ] 23:05:33 DEBUG } 23:05:33 DEBUG } 23:05:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391534" 23:05:34 INFO [http_cmd]: http status code OK 23:05:34 DEBUG --- status code --- 23:05:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:34 DEBUG --- http response --- 23:05:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:34 DEBUG ----- output ----- 23:05:34 DEBUG { 23:05:34 DEBUG "status": "success", 23:05:34 DEBUG "data": { 23:05:34 DEBUG "resultType": "vector", 23:05:34 DEBUG "result": [ 23:05:34 DEBUG { 23:05:34 DEBUG "metric": { 23:05:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:05:34 DEBUG }, 23:05:34 DEBUG "value": [ 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG [...] 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:05:34 DEBUG }, 23:05:34 DEBUG "value": [ 23:05:34 DEBUG 1713391534, 23:05:34 DEBUG "1823.9666666666667" 23:05:34 DEBUG ] 23:05:34 DEBUG } 23:05:34 DEBUG ] 23:05:34 DEBUG } 23:05:34 DEBUG } 23:05:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391534" 23:05:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391534" 23:05:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391534" 23:05:34 INFO [http_cmd]: http status code OK 23:05:34 DEBUG --- status code --- 23:05:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:34 DEBUG --- http response --- 23:05:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:34 DEBUG ----- output ----- 23:05:34 DEBUG { 23:05:34 DEBUG "status": "success", 23:05:34 DEBUG "data": { 23:05:34 DEBUG "resultType": "vector", 23:05:34 DEBUG "result": [ 23:05:34 DEBUG { 23:05:34 DEBUG "metric": { 23:05:34 DEBUG "instance": "10.142.0.107:9100" 23:05:34 DEBUG }, 23:05:34 DEBUG "value": [ 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG [...] 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG "instance": "10.142.0.43:9100" 23:05:34 DEBUG }, 23:05:34 DEBUG "value": [ 23:05:34 DEBUG 1713391534, 23:05:34 DEBUG "0.07733333333333405" 23:05:34 DEBUG ] 23:05:34 DEBUG } 23:05:34 DEBUG ] 23:05:34 DEBUG } 23:05:34 DEBUG } 23:05:34 INFO [http_cmd]: http status code OK 23:05:34 INFO [http_cmd]: http status code OK 23:05:34 DEBUG --- status code --- 23:05:34 DEBUG --- status code --- 23:05:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:34 DEBUG --- http response --- 23:05:34 DEBUG --- http response --- 23:05:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:34 DEBUG ----- output ----- 23:05:34 DEBUG ----- output ----- 23:05:34 DEBUG { 23:05:34 DEBUG { 23:05:34 DEBUG "status": "success", 23:05:34 DEBUG "status": "success", 23:05:34 DEBUG "data": { 23:05:34 DEBUG "data": { 23:05:34 DEBUG "resultType": "vector", 23:05:34 DEBUG "resultType": "vector", 23:05:34 DEBUG "result": [ 23:05:34 DEBUG "result": [ 23:05:34 DEBUG { 23:05:34 DEBUG { 23:05:34 DEBUG "metric": { 23:05:34 DEBUG "metric": { 23:05:34 DEBUG "instance": "10.142.0.107:9100" 23:05:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:05:34 DEBUG }, 23:05:34 DEBUG }, 23:05:34 DEBUG "value": [ 23:05:34 DEBUG "value": [ 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG [...] 23:05:34 DEBUG [...] 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG "instance": "10.142.0.43:9100" 23:05:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:05:34 DEBUG }, 23:05:34 DEBUG }, 23:05:34 DEBUG "value": [ 23:05:34 DEBUG "value": [ 23:05:34 DEBUG 1713391534, 23:05:34 DEBUG 1713391534, 23:05:34 DEBUG "60916354.53333333" 23:05:34 DEBUG "2290925568" 23:05:34 DEBUG ] 23:05:34 DEBUG ] 23:05:34 DEBUG } 23:05:34 DEBUG } 23:05:34 DEBUG ] 23:05:34 DEBUG ] 23:05:34 DEBUG } 23:05:34 DEBUG } 23:05:34 DEBUG } 23:05:34 DEBUG } 23:05:34 INFO 23:05:34 INFO [loop_until]: kubectl --namespace=xlou top pods 23:05:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:05:34 INFO [loop_until]: OK (rc = 0) 23:05:34 DEBUG --- stdout --- 23:05:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12000m 8914Mi am-b59c765b7-bgsxp 11191m 8770Mi am-b59c765b7-ch5q4 11833m 8536Mi ds-cts-0 12535m 7895Mi ds-cts-1 11975m 7757Mi ds-cts-2 12749m 7988Mi ds-idrepo-0 5251m 3220Mi ds-idrepo-1 5389m 2879Mi ds-idrepo-2 5333m 2933Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6405m 2090Mi 23:05:34 DEBUG --- stderr --- 23:05:34 DEBUG 23:05:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391534" 23:05:34 INFO [http_cmd]: http status code OK 23:05:34 DEBUG --- status code --- 23:05:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:34 DEBUG --- http response --- 23:05:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:05:34 DEBUG ----- output ----- 23:05:34 DEBUG { 23:05:34 DEBUG "status": "success", 23:05:34 DEBUG "data": { 23:05:34 DEBUG "resultType": "vector", 23:05:34 DEBUG "result": [ 23:05:34 DEBUG { 23:05:34 DEBUG "metric": { 23:05:34 DEBUG "pod": "ds-cts-0" 23:05:34 DEBUG }, 23:05:34 DEBUG "value": [ 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG [...] 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG "pod": "ds-cts-2" 23:05:34 DEBUG }, 23:05:34 DEBUG "value": [ 23:05:34 DEBUG 1713391534, 23:05:34 DEBUG "29.933333333333334" 23:05:34 DEBUG ] 23:05:34 DEBUG } 23:05:34 DEBUG ] 23:05:34 DEBUG } 23:05:34 DEBUG } 23:05:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391534" 23:05:34 INFO [http_cmd]: http status code OK 23:05:34 DEBUG --- status code --- 23:05:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:34 DEBUG --- http response --- 23:05:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:34 DEBUG ----- output ----- 23:05:34 DEBUG { 23:05:34 DEBUG "status": "success", 23:05:34 DEBUG "data": { 23:05:34 DEBUG "resultType": "vector", 23:05:34 DEBUG "result": [ 23:05:34 DEBUG { 23:05:34 DEBUG "metric": { 23:05:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:05:34 DEBUG }, 23:05:34 DEBUG "value": [ 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG [...] 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG 23:05:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:05:34 DEBUG }, 23:05:34 DEBUG "value": [ 23:05:34 DEBUG 1713391534, 23:05:34 DEBUG "1823.1666666666667" 23:05:34 DEBUG ] 23:05:34 DEBUG } 23:05:34 DEBUG ] 23:05:34 DEBUG } 23:05:34 DEBUG } 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:05:35 DEBUG { 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG { 23:05:35 DEBUG "data": { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "result": [ 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG { 23:05:35 DEBUG "result": [ 23:05:35 DEBUG "metric": { 23:05:35 DEBUG { 23:05:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:05:35 DEBUG "metric": { 23:05:35 DEBUG }, 23:05:35 DEBUG "pod": "ds-idrepo-0" 23:05:35 DEBUG "value": [ 23:05:35 DEBUG }, 23:05:35 DEBUG 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:05:35 DEBUG 23:05:35 DEBUG }, 23:05:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:05:35 DEBUG "value": [ 23:05:35 DEBUG }, 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG "1824.1333333333334" 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG ] 23:05:35 DEBUG "5500.584417199273" 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 INFO 23:05:35 INFO [loop_until]: kubectl --namespace=xlou top node 23:05:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391535" 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:35 DEBUG --- http response --- 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391535" 23:05:35 DEBUG ----- output ----- 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391535" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391535" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391535" 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG { 23:05:35 DEBUG ----- output ----- 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG { 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG "status": "success", 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG { 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG "data": { 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:35 DEBUG "data": { 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG --- http response --- 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG "result": [ 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:35 DEBUG "data": { 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:35 DEBUG { 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 INFO [loop_until]: OK (rc = 0) 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG { 23:05:35 DEBUG "result": [ 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG { 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG "metric": { 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG { 23:05:35 DEBUG --- stdout --- 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG "result": [ 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG { 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG { 23:05:35 DEBUG { 23:05:35 DEBUG { 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG { 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG { 23:05:35 DEBUG "pod": "ds-idrepo-0" 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5447Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5279m 17% 6157Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11847m 39% 11113Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 2993Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12905m 43% 10158Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 4972Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11942m 40% 11526Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27909m 93% 2914Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11794m 39% 10614Mi 9% gke-xlou-cdm-ds-0f601751-j488 5526m 18% 5517Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11892m 39% 11321Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5799m 19% 5995Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13009m 43% 9674Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6742m 42% 5869Mi 10% 23:05:35 DEBUG { 23:05:35 DEBUG { 23:05:35 DEBUG "data": { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:05:35 DEBUG "data": { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG }, 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG --- stderr --- 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "pod": "ds-idrepo-0" 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "data": { 23:05:35 DEBUG "data": { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "value": [ 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG 23:05:35 DEBUG "data": { 23:05:35 DEBUG "pod": "ds-idrepo-0" 23:05:35 DEBUG "result": [ 23:05:35 DEBUG }, 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG { 23:05:35 DEBUG "result": [ 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG }, 23:05:35 DEBUG { 23:05:35 DEBUG "value": [ 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG { 23:05:35 DEBUG "result": [ 23:05:35 DEBUG 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "result": [ 23:05:35 DEBUG "value": [ 23:05:35 DEBUG "metric": { 23:05:35 DEBUG 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "result": [ 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "data": { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG { 23:05:35 DEBUG 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG { 23:05:35 DEBUG 23:05:35 DEBUG "pod": "ds-idrepo-0" 23:05:35 DEBUG 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG { 23:05:35 DEBUG "instance": "10.142.0.107:9100" 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "pod": "ds-cts-0" 23:05:35 DEBUG "metric": { 23:05:35 DEBUG [...] 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "instance": "10.142.0.107:9100" 23:05:35 DEBUG "metric": { 23:05:35 DEBUG 23:05:35 DEBUG }, 23:05:35 DEBUG 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:05:35 DEBUG "metric": { 23:05:35 DEBUG }, 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:05:35 DEBUG "result": [ 23:05:35 DEBUG }, 23:05:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:05:35 DEBUG 23:05:35 DEBUG "instance": "10.142.0.107:9100" 23:05:35 DEBUG }, 23:05:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:05:35 DEBUG 23:05:35 DEBUG "value": [ 23:05:35 DEBUG [...] 23:05:35 DEBUG "pod": "ds-idrepo-0" 23:05:35 DEBUG }, 23:05:35 DEBUG "pod": "ds-idrepo-0" 23:05:35 DEBUG "value": [ 23:05:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:05:35 DEBUG }, 23:05:35 DEBUG { 23:05:35 DEBUG "value": [ 23:05:35 DEBUG }, 23:05:35 DEBUG 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG }, 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG }, 23:05:35 DEBUG 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG "metric": { 23:05:35 DEBUG 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG "pod": "ds-cts-0" 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG }, 23:05:35 DEBUG 23:05:35 DEBUG "pod": "ds-idrepo-2" 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "value": [ 23:05:35 DEBUG [...] 23:05:35 DEBUG }, 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:05:35 DEBUG 23:05:35 DEBUG }, 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG }, 23:05:35 DEBUG 23:05:35 DEBUG "value": [ 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "0" 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG "0" 23:05:35 DEBUG ] 23:05:35 DEBUG "pod": "ds-idrepo-2" 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "instance": "10.142.0.43:9100" 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG "pod": "ds-cts-2" 23:05:35 DEBUG ] 23:05:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:05:35 DEBUG 23:05:35 DEBUG "instance": "10.142.0.43:9100" 23:05:35 DEBUG 23:05:35 DEBUG "407.45069757031655" 23:05:35 DEBUG 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG }, 23:05:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:05:35 DEBUG }, 23:05:35 DEBUG 23:05:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:05:35 DEBUG 23:05:35 DEBUG }, 23:05:35 DEBUG } 23:05:35 DEBUG }, 23:05:35 DEBUG "instance": "10.142.0.43:9100" 23:05:35 DEBUG }, 23:05:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:05:35 DEBUG ] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "pod": "ds-idrepo-2" 23:05:35 DEBUG "value": [ 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:05:35 DEBUG }, 23:05:35 DEBUG 23:05:35 DEBUG "value": [ 23:05:35 DEBUG ] 23:05:35 DEBUG "value": [ 23:05:35 DEBUG "value": [ 23:05:35 DEBUG }, 23:05:35 DEBUG }, 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG "pod": "ds-cts-2" 23:05:35 DEBUG }, 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG } 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG "value": [ 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG "0" 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG "0.04933333333333354" 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG "1824.2666666666667" 23:05:35 DEBUG "0.00012976444444444445" 23:05:35 DEBUG } 23:05:35 DEBUG "6.621745092397096" 23:05:35 DEBUG "216740449280" 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG } 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG ] 23:05:35 DEBUG "0.018" 23:05:35 DEBUG ] 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG ] 23:05:35 DEBUG "pod": "ds-cts-2" 23:05:35 DEBUG ] 23:05:35 DEBUG ] 23:05:35 DEBUG ] 23:05:35 DEBUG "60917465.099999994" 23:05:35 DEBUG "0" 23:05:35 DEBUG } 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG "0" 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG "329447.5" 23:05:35 DEBUG } 23:05:35 DEBUG }, 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG ] 23:05:35 DEBUG "0" 23:05:35 DEBUG ] 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG ] 23:05:35 DEBUG ] 23:05:35 DEBUG "value": [ 23:05:35 DEBUG ] 23:05:35 DEBUG ] 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG 1713391535, 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG "0" 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:06:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391593" 23:06:33 INFO [http_cmd]: http status code OK 23:06:33 DEBUG --- status code --- 23:06:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:33 DEBUG --- http response --- 23:06:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:06:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391593" 23:06:33 ERROR [http_cmd]: ERROR 23:06:33 ERROR --- status code --- 23:06:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:06:33 ERROR --- http response --- 23:06:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:06:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:06:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391594" 23:06:34 INFO [http_cmd]: http status code OK 23:06:34 DEBUG --- status code --- 23:06:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:34 DEBUG --- http response --- 23:06:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:34 DEBUG ----- output ----- 23:06:34 DEBUG { 23:06:34 DEBUG "status": "success", 23:06:34 DEBUG "data": { 23:06:34 DEBUG "resultType": "vector", 23:06:34 DEBUG "result": [ 23:06:34 DEBUG { 23:06:34 DEBUG "metric": { 23:06:34 DEBUG "pod": "ds-idrepo-0" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG [...] 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG "pod": "ds-idrepo-2" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 1713391594, 23:06:34 DEBUG "5.242499787234085" 23:06:34 DEBUG ] 23:06:34 DEBUG } 23:06:34 DEBUG ] 23:06:34 DEBUG } 23:06:34 DEBUG } 23:06:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391594" 23:06:34 INFO [http_cmd]: http status code OK 23:06:34 DEBUG --- status code --- 23:06:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:34 DEBUG --- http response --- 23:06:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:34 DEBUG ----- output ----- 23:06:34 DEBUG { 23:06:34 DEBUG "status": "success", 23:06:34 DEBUG "data": { 23:06:34 DEBUG "resultType": "vector", 23:06:34 DEBUG "result": [ 23:06:34 DEBUG { 23:06:34 DEBUG "metric": { 23:06:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG [...] 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 1713391594, 23:06:34 DEBUG "1811.3666666666666" 23:06:34 DEBUG ] 23:06:34 DEBUG } 23:06:34 DEBUG ] 23:06:34 DEBUG } 23:06:34 DEBUG } 23:06:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391594" 23:06:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391594" 23:06:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391594" 23:06:34 INFO [http_cmd]: http status code OK 23:06:34 DEBUG --- status code --- 23:06:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:34 DEBUG --- http response --- 23:06:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:34 DEBUG ----- output ----- 23:06:34 DEBUG { 23:06:34 DEBUG "status": "success", 23:06:34 DEBUG "data": { 23:06:34 DEBUG "resultType": "vector", 23:06:34 DEBUG "result": [ 23:06:34 DEBUG { 23:06:34 DEBUG "metric": { 23:06:34 DEBUG "instance": "10.142.0.107:9100" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG [...] 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG "instance": "10.142.0.43:9100" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 1713391594, 23:06:34 DEBUG "0.15626666666666628" 23:06:34 DEBUG ] 23:06:34 DEBUG } 23:06:34 DEBUG ] 23:06:34 DEBUG } 23:06:34 DEBUG } 23:06:34 INFO [http_cmd]: http status code OK 23:06:34 DEBUG --- status code --- 23:06:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:34 DEBUG --- http response --- 23:06:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:34 DEBUG ----- output ----- 23:06:34 DEBUG { 23:06:34 DEBUG "status": "success", 23:06:34 DEBUG "data": { 23:06:34 DEBUG "resultType": "vector", 23:06:34 DEBUG "result": [ 23:06:34 DEBUG { 23:06:34 DEBUG "metric": { 23:06:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG [...] 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 1713391594, 23:06:34 DEBUG "2291736576" 23:06:34 DEBUG ] 23:06:34 DEBUG } 23:06:34 DEBUG ] 23:06:34 INFO [http_cmd]: http status code OK 23:06:34 DEBUG } 23:06:34 DEBUG --- status code --- 23:06:34 DEBUG } 23:06:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:34 DEBUG --- http response --- 23:06:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:34 DEBUG ----- output ----- 23:06:34 DEBUG { 23:06:34 DEBUG "status": "success", 23:06:34 DEBUG "data": { 23:06:34 DEBUG "resultType": "vector", 23:06:34 DEBUG "result": [ 23:06:34 DEBUG { 23:06:34 DEBUG "metric": { 23:06:34 DEBUG "instance": "10.142.0.107:9100" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG [...] 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG "instance": "10.142.0.43:9100" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 1713391594, 23:06:34 DEBUG "60742126.266666666" 23:06:34 DEBUG ] 23:06:34 DEBUG } 23:06:34 DEBUG ] 23:06:34 DEBUG } 23:06:34 DEBUG } 23:06:34 INFO 23:06:34 INFO [loop_until]: kubectl --namespace=xlou top pods 23:06:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:06:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391594" 23:06:34 INFO [http_cmd]: http status code OK 23:06:34 DEBUG --- status code --- 23:06:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:34 DEBUG --- http response --- 23:06:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:06:34 DEBUG ----- output ----- 23:06:34 DEBUG { 23:06:34 DEBUG "status": "success", 23:06:34 DEBUG "data": { 23:06:34 DEBUG "resultType": "vector", 23:06:34 DEBUG "result": [ 23:06:34 DEBUG { 23:06:34 DEBUG "metric": { 23:06:34 DEBUG "pod": "ds-cts-0" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG [...] 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG "pod": "ds-cts-2" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 1713391594, 23:06:34 DEBUG "29.966666666666665" 23:06:34 DEBUG ] 23:06:34 DEBUG } 23:06:34 DEBUG ] 23:06:34 DEBUG } 23:06:34 DEBUG } 23:06:34 INFO [loop_until]: OK (rc = 0) 23:06:34 DEBUG --- stdout --- 23:06:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12217m 8915Mi am-b59c765b7-bgsxp 11102m 8770Mi am-b59c765b7-ch5q4 11659m 8536Mi ds-cts-0 12513m 7918Mi ds-cts-1 12193m 7773Mi ds-cts-2 12855m 7955Mi ds-idrepo-0 5538m 3221Mi ds-idrepo-1 5182m 2879Mi ds-idrepo-2 5174m 2932Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6784m 2092Mi 23:06:34 DEBUG --- stderr --- 23:06:34 DEBUG 23:06:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391594" 23:06:34 INFO [http_cmd]: http status code OK 23:06:34 DEBUG --- status code --- 23:06:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:34 DEBUG --- http response --- 23:06:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:34 DEBUG ----- output ----- 23:06:34 DEBUG { 23:06:34 DEBUG "status": "success", 23:06:34 DEBUG "data": { 23:06:34 DEBUG "resultType": "vector", 23:06:34 DEBUG "result": [ 23:06:34 DEBUG { 23:06:34 DEBUG "metric": { 23:06:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG [...] 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG 23:06:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:06:34 DEBUG }, 23:06:34 DEBUG "value": [ 23:06:34 DEBUG 1713391594, 23:06:34 DEBUG "1832.3333333333333" 23:06:34 DEBUG ] 23:06:34 DEBUG } 23:06:34 DEBUG ] 23:06:34 DEBUG } 23:06:34 DEBUG } 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG { 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "data": { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "result": [ 23:06:35 DEBUG { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "pod": "ds-idrepo-0" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "pod": "ds-idrepo-2" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG "10457.872340425532" 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG ] 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG } 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:35 DEBUG ] 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG } 23:06:35 DEBUG { 23:06:35 DEBUG } 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "data": { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "result": [ 23:06:35 DEBUG { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG "1810.5" 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 INFO 23:06:35 INFO [loop_until]: kubectl --namespace=xlou top node 23:06:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391595" 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG --- status code --- 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG --- status code --- 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG --- status code --- 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- status code --- 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- status code --- 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:35 DEBUG { 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG { 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "data": { 23:06:35 DEBUG { 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG { 23:06:35 DEBUG { 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG { 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG { 23:06:35 DEBUG { 23:06:35 DEBUG { 23:06:35 DEBUG { 23:06:35 DEBUG { 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG { 23:06:35 DEBUG "data": { 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:35 DEBUG { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG { 23:06:35 DEBUG "data": { 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "data": { 23:06:35 DEBUG "data": { 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "data": { 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "result": [ 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "data": { 23:06:35 DEBUG "data": { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "data": { 23:06:35 DEBUG "data": { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "data": { 23:06:35 DEBUG "result": [ 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG { 23:06:35 DEBUG "data": { 23:06:35 DEBUG { 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG { 23:06:35 DEBUG "result": [ 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "result": [ 23:06:35 DEBUG "data": { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "result": [ 23:06:35 DEBUG "result": [ 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG { 23:06:35 DEBUG "result": [ 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "data": { 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "data": { 23:06:35 DEBUG "result": [ 23:06:35 DEBUG { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "result": [ 23:06:35 DEBUG "result": [ 23:06:35 DEBUG { 23:06:35 DEBUG { 23:06:35 DEBUG "result": [ 23:06:35 DEBUG "metric": { 23:06:35 DEBUG { 23:06:35 DEBUG "data": { 23:06:35 DEBUG "result": [ 23:06:35 DEBUG "pod": "ds-idrepo-0" 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG { 23:06:35 DEBUG { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "result": [ 23:06:35 DEBUG { 23:06:35 DEBUG { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG { 23:06:35 DEBUG "pod": "ds-idrepo-0" 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG { 23:06:35 DEBUG }, 23:06:35 DEBUG "result": [ 23:06:35 DEBUG "result": [ 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "pod": "ds-cts-0" 23:06:35 DEBUG { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "instance": "10.142.0.107:9100" 23:06:35 DEBUG "instance": "10.142.0.107:9100" 23:06:35 DEBUG "metric": { 23:06:35 DEBUG }, 23:06:35 DEBUG "pod": "ds-idrepo-0" 23:06:35 DEBUG "result": [ 23:06:35 INFO [loop_until]: OK (rc = 0) 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "value": [ 23:06:35 DEBUG { 23:06:35 DEBUG { 23:06:35 DEBUG "pod": "ds-idrepo-0" 23:06:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:06:35 DEBUG }, 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "pod": "ds-idrepo-0" 23:06:35 DEBUG }, 23:06:35 DEBUG }, 23:06:35 DEBUG }, 23:06:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:06:35 DEBUG "value": [ 23:06:35 DEBUG }, 23:06:35 DEBUG { 23:06:35 DEBUG --- stdout --- 23:06:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:06:35 DEBUG 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG }, 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG "instance": "10.142.0.107:9100" 23:06:35 DEBUG "value": [ 23:06:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:06:35 DEBUG "value": [ 23:06:35 DEBUG "value": [ 23:06:35 DEBUG }, 23:06:35 DEBUG 23:06:35 DEBUG "value": [ 23:06:35 DEBUG "metric": { 23:06:35 DEBUG }, 23:06:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5438Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5301m 17% 6151Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11822m 39% 11127Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 2991Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12267m 41% 10152Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 177m 0% 4985Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12155m 40% 11527Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28078m 94% 2922Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11717m 39% 10607Mi 9% gke-xlou-cdm-ds-0f601751-j488 5646m 18% 5514Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11696m 39% 11330Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5827m 19% 5988Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13192m 44% 9674Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6783m 42% 5866Mi 10% 23:06:35 DEBUG 23:06:35 DEBUG "pod": "ds-cts-0" 23:06:35 DEBUG "pod": "ds-idrepo-0" 23:06:35 DEBUG "value": [ 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG }, 23:06:35 DEBUG 23:06:35 DEBUG }, 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:06:35 DEBUG "value": [ 23:06:35 DEBUG --- stderr --- 23:06:35 DEBUG 23:06:35 DEBUG }, 23:06:35 DEBUG }, 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG }, 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG "pod": "ds-idrepo-2" 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG }, 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "pod": "ds-cts-2" 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:06:35 DEBUG "pod": "ds-idrepo-2" 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG }, 23:06:35 DEBUG 23:06:35 DEBUG "pod": "ds-idrepo-2" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG "instance": "10.142.0.43:9100" 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG "0" 23:06:35 DEBUG [...] 23:06:35 DEBUG }, 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG "instance": "10.142.0.43:9100" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG "pod": "ds-idrepo-2" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "pod": "ds-idrepo-2" 23:06:35 DEBUG "4108.449848024316" 23:06:35 DEBUG 23:06:35 DEBUG }, 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "0" 23:06:35 DEBUG }, 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG "219996818432" 23:06:35 DEBUG ] 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG 23:06:35 DEBUG }, 23:06:35 DEBUG ] 23:06:35 DEBUG 23:06:35 DEBUG "value": [ 23:06:35 DEBUG "0.00008631333333333332" 23:06:35 DEBUG "instance": "10.142.0.43:9100" 23:06:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:06:35 DEBUG ] 23:06:35 DEBUG "value": [ 23:06:35 DEBUG ] 23:06:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:06:35 DEBUG } 23:06:35 DEBUG "0" 23:06:35 DEBUG 23:06:35 DEBUG "6.640413488281495" 23:06:35 DEBUG 23:06:35 DEBUG "value": [ 23:06:35 DEBUG } 23:06:35 DEBUG "pod": "ds-cts-2" 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG ] 23:06:35 DEBUG }, 23:06:35 DEBUG }, 23:06:35 DEBUG } 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG } 23:06:35 DEBUG }, 23:06:35 DEBUG ] 23:06:35 DEBUG ] 23:06:35 DEBUG 23:06:35 DEBUG ] 23:06:35 DEBUG "pod": "ds-cts-2" 23:06:35 DEBUG }, 23:06:35 DEBUG ] 23:06:35 DEBUG }, 23:06:35 DEBUG "1810.3666666666666" 23:06:35 DEBUG } 23:06:35 DEBUG "value": [ 23:06:35 DEBUG "value": [ 23:06:35 DEBUG ] 23:06:35 DEBUG "0.08633333333333322" 23:06:35 DEBUG ] 23:06:35 DEBUG "value": [ 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:06:35 DEBUG } 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG } 23:06:35 DEBUG "value": [ 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG }, 23:06:35 DEBUG ] 23:06:35 DEBUG "0" 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG } 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG ] 23:06:35 DEBUG ] 23:06:35 DEBUG "60742683.233333334" 23:06:35 DEBUG "0.018" 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG "328535" 23:06:35 DEBUG } 23:06:35 DEBUG "value": [ 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG "0" 23:06:35 DEBUG "0" 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG ] 23:06:35 DEBUG ] 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG 1713391595, 23:06:35 DEBUG ] 23:06:35 DEBUG ] 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG "0" 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG ] 23:06:35 DEBUG ] 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:07:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391653" 23:07:33 INFO [http_cmd]: http status code OK 23:07:33 DEBUG --- status code --- 23:07:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:33 DEBUG --- http response --- 23:07:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:07:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391653" 23:07:33 ERROR [http_cmd]: ERROR 23:07:33 ERROR --- status code --- 23:07:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:07:33 ERROR --- http response --- 23:07:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:07:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:07:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391654" 23:07:34 INFO [http_cmd]: http status code OK 23:07:34 DEBUG --- status code --- 23:07:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:34 DEBUG --- http response --- 23:07:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:07:34 DEBUG ----- output ----- 23:07:34 DEBUG { 23:07:34 DEBUG "status": "success", 23:07:34 DEBUG "data": { 23:07:34 DEBUG "resultType": "vector", 23:07:34 DEBUG "result": [ 23:07:34 DEBUG { 23:07:34 DEBUG "metric": { 23:07:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:07:34 DEBUG }, 23:07:34 DEBUG "value": [ 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG [...] 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:07:34 DEBUG }, 23:07:34 DEBUG "value": [ 23:07:34 DEBUG 1713391654, 23:07:34 DEBUG "0.00002304241668387234" 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG } 23:07:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391654" 23:07:34 INFO [http_cmd]: http status code OK 23:07:34 DEBUG --- status code --- 23:07:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:34 DEBUG --- http response --- 23:07:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:34 DEBUG ----- output ----- 23:07:34 DEBUG { 23:07:34 DEBUG "status": "success", 23:07:34 DEBUG "data": { 23:07:34 DEBUG "resultType": "vector", 23:07:34 DEBUG "result": [ 23:07:34 DEBUG { 23:07:34 DEBUG "metric": { 23:07:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:07:34 DEBUG }, 23:07:34 DEBUG "value": [ 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG [...] 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:07:34 DEBUG }, 23:07:34 DEBUG "value": [ 23:07:34 DEBUG 1713391654, 23:07:34 DEBUG "1833.6666666666667" 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG } 23:07:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391654" 23:07:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391654" 23:07:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391654" 23:07:34 INFO [http_cmd]: http status code OK 23:07:34 DEBUG --- status code --- 23:07:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:34 DEBUG --- http response --- 23:07:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:34 DEBUG ----- output ----- 23:07:34 DEBUG { 23:07:34 DEBUG "status": "success", 23:07:34 DEBUG "data": { 23:07:34 DEBUG "resultType": "vector", 23:07:34 DEBUG "result": [ 23:07:34 DEBUG { 23:07:34 DEBUG "metric": { 23:07:34 DEBUG "instance": "10.142.0.107:9100" 23:07:34 DEBUG }, 23:07:34 DEBUG "value": [ 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG [...] 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG "instance": "10.142.0.43:9100" 23:07:34 DEBUG }, 23:07:34 DEBUG "value": [ 23:07:34 DEBUG 1713391654, 23:07:34 DEBUG "0.16416666666666704" 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG } 23:07:34 INFO [http_cmd]: http status code OK 23:07:34 DEBUG --- status code --- 23:07:34 INFO [http_cmd]: http status code OK 23:07:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:34 DEBUG --- status code --- 23:07:34 DEBUG --- http response --- 23:07:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:34 DEBUG --- http response --- 23:07:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:34 DEBUG ----- output ----- 23:07:34 DEBUG ----- output ----- 23:07:34 DEBUG { 23:07:34 DEBUG { 23:07:34 DEBUG "status": "success", 23:07:34 DEBUG "status": "success", 23:07:34 DEBUG "data": { 23:07:34 DEBUG "data": { 23:07:34 DEBUG "resultType": "vector", 23:07:34 DEBUG "resultType": "vector", 23:07:34 DEBUG "result": [ 23:07:34 DEBUG "result": [ 23:07:34 DEBUG { 23:07:34 DEBUG { 23:07:34 DEBUG "metric": { 23:07:34 DEBUG "metric": { 23:07:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:07:34 DEBUG "instance": "10.142.0.107:9100" 23:07:34 DEBUG }, 23:07:34 DEBUG }, 23:07:34 DEBUG "value": [ 23:07:34 DEBUG "value": [ 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG [...] 23:07:34 DEBUG [...] 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG "instance": "10.142.0.43:9100" 23:07:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:07:34 DEBUG }, 23:07:34 DEBUG }, 23:07:34 DEBUG "value": [ 23:07:34 DEBUG "value": [ 23:07:34 DEBUG 1713391654, 23:07:34 DEBUG 1713391654, 23:07:34 DEBUG "61141071.26666666" 23:07:34 DEBUG "2291638272" 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG } 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG } 23:07:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391654" 23:07:34 INFO [http_cmd]: http status code OK 23:07:34 DEBUG --- status code --- 23:07:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:34 DEBUG --- http response --- 23:07:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:07:34 DEBUG ----- output ----- 23:07:34 DEBUG { 23:07:34 DEBUG "status": "success", 23:07:34 DEBUG "data": { 23:07:34 DEBUG "resultType": "vector", 23:07:34 DEBUG "result": [ 23:07:34 DEBUG { 23:07:34 DEBUG "metric": { 23:07:34 DEBUG "pod": "ds-cts-0" 23:07:34 DEBUG }, 23:07:34 DEBUG "value": [ 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG [...] 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG "pod": "ds-cts-2" 23:07:34 DEBUG }, 23:07:34 DEBUG "value": [ 23:07:34 DEBUG 1713391654, 23:07:34 DEBUG "30" 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG } 23:07:34 INFO 23:07:34 INFO [loop_until]: kubectl --namespace=xlou top pods 23:07:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:07:34 INFO [loop_until]: OK (rc = 0) 23:07:34 DEBUG --- stdout --- 23:07:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12024m 8917Mi am-b59c765b7-bgsxp 11163m 8770Mi am-b59c765b7-ch5q4 11862m 8536Mi ds-cts-0 12542m 7906Mi ds-cts-1 11748m 7750Mi ds-cts-2 13101m 7970Mi ds-idrepo-0 5259m 3221Mi ds-idrepo-1 5197m 2879Mi ds-idrepo-2 5205m 2932Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 3m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6739m 2094Mi 23:07:34 DEBUG --- stderr --- 23:07:34 DEBUG 23:07:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391654" 23:07:34 INFO [http_cmd]: http status code OK 23:07:34 DEBUG --- status code --- 23:07:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:34 DEBUG --- http response --- 23:07:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:34 DEBUG ----- output ----- 23:07:34 DEBUG { 23:07:34 DEBUG "status": "success", 23:07:34 DEBUG "data": { 23:07:34 DEBUG "resultType": "vector", 23:07:34 DEBUG "result": [ 23:07:34 DEBUG { 23:07:34 DEBUG "metric": { 23:07:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:07:34 DEBUG }, 23:07:34 DEBUG "value": [ 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG [...] 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG 23:07:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:07:34 DEBUG }, 23:07:34 DEBUG "value": [ 23:07:34 DEBUG 1713391654, 23:07:34 DEBUG "1819.5666666666666" 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG ] 23:07:34 DEBUG } 23:07:34 DEBUG } 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "data": { 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "result": [ 23:07:35 DEBUG "result": [ 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:07:35 DEBUG "pod": "ds-idrepo-0" 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:07:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG "1834.2" 23:07:35 DEBUG "0" 23:07:35 DEBUG ] 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391655" 23:07:35 INFO 23:07:35 INFO [loop_until]: kubectl --namespace=xlou top node 23:07:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "result": [ 23:07:35 DEBUG { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "instance": "10.142.0.107:9100" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "instance": "10.142.0.43:9100" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG "223306249216" 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391655" 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG --- status code --- 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG --- status code --- 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG --- status code --- 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "data": { 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "data": { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "data": { 23:07:35 DEBUG "data": { 23:07:35 DEBUG "data": { 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "data": { 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "data": { 23:07:35 DEBUG "data": { 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG "result": [ 23:07:35 DEBUG "result": [ 23:07:35 DEBUG "data": { 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "result": [ 23:07:35 DEBUG "result": [ 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "result": [ 23:07:35 DEBUG "result": [ 23:07:35 DEBUG "result": [ 23:07:35 DEBUG "result": [ 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "result": [ 23:07:35 DEBUG "result": [ 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG "result": [ 23:07:35 DEBUG "result": [ 23:07:35 DEBUG "result": [ 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "result": [ 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "data": { 23:07:35 DEBUG "pod": "ds-idrepo-0" 23:07:35 DEBUG "pod": "ds-cts-0" 23:07:35 DEBUG { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:07:35 DEBUG "instance": "10.142.0.107:9100" 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "pod": "ds-idrepo-0" 23:07:35 DEBUG "pod": "ds-idrepo-0" 23:07:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:07:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "pod": "ds-idrepo-0" 23:07:35 DEBUG "pod": "ds-idrepo-0" 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG "pod": "ds-cts-0" 23:07:35 DEBUG "instance": "10.142.0.107:9100" 23:07:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG "result": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "pod": "ds-idrepo-0" 23:07:35 INFO [loop_until]: OK (rc = 0) 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG { 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG }, 23:07:35 DEBUG --- stdout --- 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "metric": { 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "value": [ 23:07:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5435Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5299m 17% 6157Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11855m 39% 11132Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3001Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12336m 41% 10151Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 134m 0% 4976Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12014m 40% 11516Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28105m 94% 2918Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11842m 39% 10615Mi 9% gke-xlou-cdm-ds-0f601751-j488 5613m 18% 5519Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11777m 39% 11338Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5853m 19% 5993Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13176m 44% 9694Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6775m 42% 5864Mi 9% 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG --- stderr --- 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG }, 23:07:35 DEBUG [...] 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG [...] 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG [...] 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:07:35 DEBUG "instance": "10.142.0.43:9100" 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG }, 23:07:35 DEBUG 23:07:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:07:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:07:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:07:35 DEBUG "pod": "ds-cts-2" 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "pod": "ds-idrepo-2" 23:07:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG "pod": "ds-cts-2" 23:07:35 DEBUG "instance": "10.142.0.43:9100" 23:07:35 DEBUG "pod": "ds-idrepo-2" 23:07:35 DEBUG }, 23:07:35 DEBUG "pod": "ds-idrepo-2" 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG 23:07:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG }, 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG "pod": "ds-cts-2" 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG "0" 23:07:35 DEBUG }, 23:07:35 DEBUG "0.08599999999999973" 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG "61140305.53333333" 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG "0" 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG "1834.8" 23:07:35 DEBUG "6.624793285239178" 23:07:35 DEBUG "0" 23:07:35 DEBUG ] 23:07:35 DEBUG "0" 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG "0.018" 23:07:35 DEBUG ] 23:07:35 DEBUG "value": [ 23:07:35 DEBUG ] 23:07:35 DEBUG "0" 23:07:35 DEBUG ] 23:07:35 DEBUG "328444" 23:07:35 DEBUG ] 23:07:35 DEBUG "0" 23:07:35 DEBUG ] 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG "0.00015596" 23:07:35 DEBUG ] 23:07:35 DEBUG "0" 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG 1713391655, 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG "0" 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 DEBUG } 23:08:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391713" 23:08:33 INFO [http_cmd]: http status code OK 23:08:33 DEBUG --- status code --- 23:08:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:33 DEBUG --- http response --- 23:08:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:08:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391713" 23:08:33 ERROR [http_cmd]: ERROR 23:08:33 ERROR --- status code --- 23:08:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:08:33 ERROR --- http response --- 23:08:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:08:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:08:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391714" 23:08:34 INFO [http_cmd]: http status code OK 23:08:34 DEBUG --- status code --- 23:08:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:34 DEBUG --- http response --- 23:08:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:34 DEBUG ----- output ----- 23:08:34 DEBUG { 23:08:34 DEBUG "status": "success", 23:08:34 DEBUG "data": { 23:08:34 DEBUG "resultType": "vector", 23:08:34 DEBUG "result": [ 23:08:34 DEBUG { 23:08:34 DEBUG "metric": { 23:08:34 DEBUG "pod": "ds-idrepo-0" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG [...] 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 1713391714, 23:08:34 DEBUG "0" 23:08:34 DEBUG ] 23:08:34 DEBUG } 23:08:34 DEBUG ] 23:08:34 DEBUG } 23:08:34 DEBUG } 23:08:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391714" 23:08:34 INFO [http_cmd]: http status code OK 23:08:34 DEBUG --- status code --- 23:08:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:34 DEBUG --- http response --- 23:08:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:34 DEBUG ----- output ----- 23:08:34 DEBUG { 23:08:34 DEBUG "status": "success", 23:08:34 DEBUG "data": { 23:08:34 DEBUG "resultType": "vector", 23:08:34 DEBUG "result": [ 23:08:34 DEBUG { 23:08:34 DEBUG "metric": { 23:08:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG [...] 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 1713391714, 23:08:34 DEBUG "1823" 23:08:34 DEBUG ] 23:08:34 DEBUG } 23:08:34 DEBUG ] 23:08:34 DEBUG } 23:08:34 DEBUG } 23:08:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391714" 23:08:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391714" 23:08:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391714" 23:08:34 INFO [http_cmd]: http status code OK 23:08:34 DEBUG --- status code --- 23:08:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:34 DEBUG --- http response --- 23:08:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:34 DEBUG ----- output ----- 23:08:34 DEBUG { 23:08:34 DEBUG "status": "success", 23:08:34 DEBUG "data": { 23:08:34 DEBUG "resultType": "vector", 23:08:34 DEBUG "result": [ 23:08:34 DEBUG { 23:08:34 DEBUG "metric": { 23:08:34 DEBUG "instance": "10.142.0.107:9100" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG [...] 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG "instance": "10.142.0.43:9100" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 1713391714, 23:08:34 DEBUG "0.0679000000000002" 23:08:34 DEBUG ] 23:08:34 DEBUG } 23:08:34 DEBUG ] 23:08:34 DEBUG } 23:08:34 DEBUG } 23:08:34 INFO [http_cmd]: http status code OK 23:08:34 DEBUG --- status code --- 23:08:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:34 DEBUG --- http response --- 23:08:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:34 DEBUG ----- output ----- 23:08:34 DEBUG { 23:08:34 DEBUG "status": "success", 23:08:34 DEBUG "data": { 23:08:34 DEBUG "resultType": "vector", 23:08:34 DEBUG "result": [ 23:08:34 DEBUG { 23:08:34 DEBUG "metric": { 23:08:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG [...] 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 1713391714, 23:08:34 DEBUG "2293121024" 23:08:34 DEBUG ] 23:08:34 DEBUG } 23:08:34 DEBUG ] 23:08:34 INFO [http_cmd]: http status code OK 23:08:34 DEBUG } 23:08:34 DEBUG --- status code --- 23:08:34 DEBUG } 23:08:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:34 DEBUG --- http response --- 23:08:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:34 DEBUG ----- output ----- 23:08:34 DEBUG { 23:08:34 DEBUG "status": "success", 23:08:34 DEBUG "data": { 23:08:34 DEBUG "resultType": "vector", 23:08:34 DEBUG "result": [ 23:08:34 DEBUG { 23:08:34 DEBUG "metric": { 23:08:34 DEBUG "instance": "10.142.0.107:9100" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG [...] 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG "instance": "10.142.0.43:9100" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 1713391714, 23:08:34 DEBUG "60945986.4" 23:08:34 DEBUG ] 23:08:34 DEBUG } 23:08:34 DEBUG ] 23:08:34 DEBUG } 23:08:34 DEBUG } 23:08:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391714" 23:08:34 INFO [http_cmd]: http status code OK 23:08:34 DEBUG --- status code --- 23:08:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:34 DEBUG --- http response --- 23:08:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:34 DEBUG ----- output ----- 23:08:34 DEBUG { 23:08:34 DEBUG "status": "success", 23:08:34 DEBUG "data": { 23:08:34 DEBUG "resultType": "vector", 23:08:34 DEBUG "result": [ 23:08:34 DEBUG { 23:08:34 DEBUG "metric": { 23:08:34 DEBUG "pod": "ds-cts-0" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG [...] 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG "pod": "ds-cts-2" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 1713391714, 23:08:34 DEBUG "29.966666666666665" 23:08:34 DEBUG ] 23:08:34 DEBUG } 23:08:34 DEBUG ] 23:08:34 DEBUG } 23:08:34 DEBUG } 23:08:34 INFO 23:08:34 INFO [loop_until]: kubectl --namespace=xlou top pods 23:08:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391714" 23:08:34 INFO [http_cmd]: http status code OK 23:08:34 DEBUG --- status code --- 23:08:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:34 DEBUG --- http response --- 23:08:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:34 DEBUG ----- output ----- 23:08:34 DEBUG { 23:08:34 DEBUG "status": "success", 23:08:34 DEBUG "data": { 23:08:34 DEBUG "resultType": "vector", 23:08:34 DEBUG "result": [ 23:08:34 DEBUG { 23:08:34 DEBUG "metric": { 23:08:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG [...] 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG 23:08:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:08:34 DEBUG }, 23:08:34 DEBUG "value": [ 23:08:34 DEBUG 1713391714, 23:08:34 DEBUG "1823.8666666666666" 23:08:34 DEBUG ] 23:08:34 DEBUG } 23:08:34 DEBUG ] 23:08:34 DEBUG } 23:08:34 DEBUG } 23:08:34 INFO [loop_until]: OK (rc = 0) 23:08:34 DEBUG --- stdout --- 23:08:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12217m 8916Mi am-b59c765b7-bgsxp 10811m 8770Mi am-b59c765b7-ch5q4 12198m 8536Mi ds-cts-0 13031m 7899Mi ds-cts-1 11323m 7752Mi ds-cts-2 13025m 7978Mi ds-idrepo-0 5469m 3220Mi ds-idrepo-1 5288m 2879Mi ds-idrepo-2 5147m 2932Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6639m 2095Mi 23:08:34 DEBUG --- stderr --- 23:08:34 DEBUG 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: http status code OK 23:08:35 DEBUG --- status code --- 23:08:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:35 DEBUG --- http response --- 23:08:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:08:35 DEBUG ----- output ----- 23:08:35 DEBUG { 23:08:35 DEBUG "status": "success", 23:08:35 DEBUG "data": { 23:08:35 DEBUG "resultType": "vector", 23:08:35 DEBUG "result": [ 23:08:35 DEBUG { 23:08:35 DEBUG "metric": { 23:08:35 DEBUG "pod": "ds-idrepo-0" 23:08:35 DEBUG }, 23:08:35 DEBUG "value": [ 23:08:35 DEBUG 23:08:35 DEBUG 23:08:35 DEBUG 23:08:35 DEBUG [...] 23:08:35 DEBUG 23:08:35 DEBUG 23:08:35 DEBUG 23:08:35 DEBUG "pod": "am-b59c765b7-7xg4f" 23:08:35 DEBUG }, 23:08:35 DEBUG "value": [ 23:08:35 DEBUG 1713391715, 23:08:35 DEBUG "14261.747324708513" 23:08:35 DEBUG ] 23:08:35 DEBUG } 23:08:35 DEBUG ] 23:08:35 DEBUG } 23:08:35 DEBUG } 23:08:35 INFO [http_cmd]: http status code OK 23:08:35 DEBUG --- status code --- 23:08:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:35 DEBUG --- http response --- 23:08:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:35 DEBUG ----- output ----- 23:08:35 DEBUG { 23:08:35 DEBUG "status": "success", 23:08:35 DEBUG "data": { 23:08:35 DEBUG "resultType": "vector", 23:08:35 DEBUG "result": [ 23:08:35 DEBUG { 23:08:35 DEBUG "metric": { 23:08:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:08:35 DEBUG }, 23:08:35 DEBUG "value": [ 23:08:35 DEBUG 23:08:35 DEBUG 23:08:35 DEBUG 23:08:35 DEBUG [...] 23:08:35 DEBUG 23:08:35 DEBUG 23:08:35 DEBUG 23:08:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:08:35 DEBUG }, 23:08:35 DEBUG "value": [ 23:08:35 DEBUG 1713391715, 23:08:35 DEBUG "1822.8333333333333" 23:08:35 DEBUG ] 23:08:35 DEBUG } 23:08:35 DEBUG ] 23:08:35 DEBUG } 23:08:35 DEBUG } 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391715" 23:08:35 INFO [http_cmd]: http status code OK 23:08:35 DEBUG --- status code --- 23:08:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:35 DEBUG --- http response --- 23:08:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:35 DEBUG ----- output ----- 23:08:35 DEBUG { 23:08:35 DEBUG "status": "success", 23:08:35 DEBUG "data": { 23:08:35 DEBUG "resultType": "vector", 23:08:35 DEBUG "result": [ 23:08:35 DEBUG { 23:08:35 DEBUG "metric": { 23:08:35 DEBUG "instance": "10.142.0.107:9100" 23:08:35 DEBUG }, 23:08:35 DEBUG "value": [ 23:08:35 DEBUG 23:08:35 DEBUG 23:08:35 DEBUG 23:08:35 DEBUG [...] 23:08:35 DEBUG 23:08:35 DEBUG 23:08:35 DEBUG 23:08:35 DEBUG "instance": "10.142.0.43:9100" 23:08:35 DEBUG }, 23:08:35 DEBUG "value": [ 23:08:35 DEBUG 1713391715, 23:08:35 DEBUG "226032206848" 23:08:35 DEBUG ] 23:08:35 DEBUG } 23:08:35 DEBUG ] 23:08:35 DEBUG } 23:08:35 DEBUG } 23:08:35 INFO 23:08:35 INFO [loop_until]: kubectl --namespace=xlou top node 23:08:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391715" 23:08:35 INFO [http_cmd]: http status code OK 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG --- status code --- 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG { 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG { 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG { 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "data": { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 INFO [loop_until]: OK (rc = 0) 23:08:36 DEBUG "metric": { 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "data": { 23:08:36 DEBUG "data": { 23:08:36 DEBUG { 23:08:36 DEBUG "data": { 23:08:36 DEBUG { 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG "data": { 23:08:36 DEBUG { 23:08:36 DEBUG "result": [ 23:08:36 DEBUG --- stdout --- 23:08:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG "result": [ 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "metric": { 23:08:36 DEBUG { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG { 23:08:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 5439Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5053m 16% 6156Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11925m 39% 11117Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 2997Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12631m 42% 10164Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 4974Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11967m 40% 11519Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27892m 93% 2915Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11744m 39% 10611Mi 9% gke-xlou-cdm-ds-0f601751-j488 5612m 18% 5517Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11639m 38% 11338Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5842m 19% 6004Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12897m 43% 9683Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6745m 42% 5865Mi 10% 23:08:36 DEBUG }, 23:08:36 DEBUG { 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG { 23:08:36 DEBUG "result": [ 23:08:36 DEBUG "result": [ 23:08:36 DEBUG "result": [ 23:08:36 DEBUG "pod": "ds-idrepo-0" 23:08:36 DEBUG "result": [ 23:08:36 DEBUG "instance": "10.142.0.107:9100" 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG "data": { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG --- stderr --- 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG { 23:08:36 DEBUG { 23:08:36 DEBUG { 23:08:36 DEBUG }, 23:08:36 DEBUG { 23:08:36 DEBUG }, 23:08:36 DEBUG "data": { 23:08:36 DEBUG { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "pod": "ds-idrepo-0" 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "data": { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "ds-idrepo-0" 23:08:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "result": [ 23:08:36 DEBUG }, 23:08:36 DEBUG 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "pod": "ds-idrepo-0" 23:08:36 DEBUG }, 23:08:36 DEBUG }, 23:08:36 DEBUG "pod": "ds-cts-0" 23:08:36 DEBUG "pod": "ds-idrepo-0" 23:08:36 DEBUG "instance": "10.142.0.107:9100" 23:08:36 DEBUG 23:08:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:08:36 DEBUG 23:08:36 DEBUG "result": [ 23:08:36 DEBUG "pod": "ds-cts-0" 23:08:36 DEBUG { 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG "result": [ 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "value": [ 23:08:36 DEBUG }, 23:08:36 DEBUG }, 23:08:36 DEBUG }, 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG 23:08:36 DEBUG { 23:08:36 DEBUG }, 23:08:36 DEBUG "metric": { 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG { 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "metric": { 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "ds-idrepo-0" 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG 23:08:36 DEBUG "value": [ 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG [...] 23:08:36 DEBUG "pod": "am-b59c765b7-7xg4f" 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "instance": "10.142.0.43:9100" 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "ds-cts-2" 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG "pod": "am-b59c765b7-7xg4f" 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG 23:08:36 DEBUG "pod": "ds-idrepo-2" 23:08:36 DEBUG }, 23:08:36 DEBUG "instance": "10.142.0.43:9100" 23:08:36 DEBUG 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG "1825.0333333333333" 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG }, 23:08:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "value": [ 23:08:36 DEBUG ] 23:08:36 DEBUG "pod": "ds-idrepo-2" 23:08:36 DEBUG 23:08:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG "pod": "ds-idrepo-2" 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG }, 23:08:36 DEBUG "60944992.86666666" 23:08:36 DEBUG "pod": "ds-cts-2" 23:08:36 DEBUG 23:08:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG } 23:08:36 DEBUG }, 23:08:36 DEBUG "pod": "ds-cts-2" 23:08:36 DEBUG }, 23:08:36 DEBUG "0" 23:08:36 DEBUG }, 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG "0" 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG ] 23:08:36 DEBUG }, 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG "523.3668743012298" 23:08:36 DEBUG ] 23:08:36 DEBUG "value": [ 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG ] 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "0" 23:08:36 DEBUG ] 23:08:36 DEBUG "0.042000000000000204" 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG } 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG } 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG } 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG "0" 23:08:36 DEBUG ] 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG }, 23:08:36 DEBUG ] 23:08:36 DEBUG "0.018" 23:08:36 DEBUG } 23:08:36 DEBUG "0" 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG "6.602507963345401" 23:08:36 DEBUG ] 23:08:36 DEBUG "0" 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG "0.00034168444444444444" 23:08:36 DEBUG "value": [ 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG ] 23:08:36 DEBUG "0" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG 1713391715, 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG "328561.5" 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:09:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391773" 23:09:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391773" 23:09:33 INFO [http_cmd]: http status code OK 23:09:33 DEBUG --- status code --- 23:09:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:33 DEBUG --- http response --- 23:09:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:09:33 ERROR [http_cmd]: ERROR 23:09:33 ERROR --- status code --- 23:09:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:09:33 ERROR --- http response --- 23:09:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:09:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:09:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391774" 23:09:34 INFO [http_cmd]: http status code OK 23:09:34 DEBUG --- status code --- 23:09:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:34 DEBUG --- http response --- 23:09:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:34 DEBUG ----- output ----- 23:09:34 DEBUG { 23:09:34 DEBUG "status": "success", 23:09:34 DEBUG "data": { 23:09:34 DEBUG "resultType": "vector", 23:09:34 DEBUG "result": [ 23:09:34 DEBUG { 23:09:34 DEBUG "metric": { 23:09:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:09:34 DEBUG }, 23:09:34 DEBUG "value": [ 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG [...] 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:09:34 DEBUG }, 23:09:34 DEBUG "value": [ 23:09:34 DEBUG 1713391774, 23:09:34 DEBUG "0" 23:09:34 DEBUG ] 23:09:34 DEBUG } 23:09:34 DEBUG ] 23:09:34 DEBUG } 23:09:34 DEBUG } 23:09:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391774" 23:09:34 INFO [http_cmd]: http status code OK 23:09:34 DEBUG --- status code --- 23:09:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:34 DEBUG --- http response --- 23:09:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:34 DEBUG ----- output ----- 23:09:34 DEBUG { 23:09:34 DEBUG "status": "success", 23:09:34 DEBUG "data": { 23:09:34 DEBUG "resultType": "vector", 23:09:34 DEBUG "result": [ 23:09:34 DEBUG { 23:09:34 DEBUG "metric": { 23:09:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:09:34 DEBUG }, 23:09:34 DEBUG "value": [ 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG [...] 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:09:34 DEBUG }, 23:09:34 DEBUG "value": [ 23:09:34 DEBUG 1713391774, 23:09:34 DEBUG "1835.3" 23:09:34 DEBUG ] 23:09:34 DEBUG } 23:09:34 DEBUG ] 23:09:34 DEBUG } 23:09:34 DEBUG } 23:09:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391774" 23:09:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391774" 23:09:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391774" 23:09:34 INFO [http_cmd]: http status code OK 23:09:34 DEBUG --- status code --- 23:09:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:34 DEBUG --- http response --- 23:09:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:34 DEBUG ----- output ----- 23:09:34 DEBUG { 23:09:34 DEBUG "status": "success", 23:09:34 DEBUG "data": { 23:09:34 DEBUG "resultType": "vector", 23:09:34 DEBUG "result": [ 23:09:34 DEBUG { 23:09:34 DEBUG "metric": { 23:09:34 DEBUG "instance": "10.142.0.107:9100" 23:09:34 DEBUG }, 23:09:34 DEBUG "value": [ 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG [...] 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG "instance": "10.142.0.43:9100" 23:09:34 DEBUG }, 23:09:34 DEBUG "value": [ 23:09:34 DEBUG 1713391774, 23:09:34 DEBUG "0.2480000000000009" 23:09:34 DEBUG ] 23:09:34 DEBUG } 23:09:34 DEBUG ] 23:09:34 DEBUG } 23:09:34 DEBUG } 23:09:34 INFO [http_cmd]: http status code OK 23:09:34 INFO [http_cmd]: http status code OK 23:09:34 DEBUG --- status code --- 23:09:34 DEBUG --- status code --- 23:09:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:34 DEBUG --- http response --- 23:09:34 DEBUG --- http response --- 23:09:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:34 DEBUG ----- output ----- 23:09:34 DEBUG ----- output ----- 23:09:34 DEBUG { 23:09:34 DEBUG { 23:09:34 DEBUG "status": "success", 23:09:34 DEBUG "status": "success", 23:09:34 DEBUG "data": { 23:09:34 DEBUG "data": { 23:09:34 DEBUG "resultType": "vector", 23:09:34 DEBUG "resultType": "vector", 23:09:34 DEBUG "result": [ 23:09:34 DEBUG "result": [ 23:09:34 DEBUG { 23:09:34 DEBUG { 23:09:34 DEBUG "metric": { 23:09:34 DEBUG "metric": { 23:09:34 DEBUG "instance": "10.142.0.107:9100" 23:09:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:09:34 DEBUG }, 23:09:34 DEBUG }, 23:09:34 DEBUG "value": [ 23:09:34 DEBUG "value": [ 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG [...] 23:09:34 DEBUG [...] 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG "instance": "10.142.0.43:9100" 23:09:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:09:34 DEBUG }, 23:09:34 DEBUG }, 23:09:34 DEBUG "value": [ 23:09:34 DEBUG "value": [ 23:09:34 DEBUG 1713391774, 23:09:34 DEBUG 1713391774, 23:09:34 DEBUG "60945150.766666666" 23:09:34 DEBUG "2297380864" 23:09:34 DEBUG ] 23:09:34 DEBUG ] 23:09:34 DEBUG } 23:09:34 DEBUG } 23:09:34 DEBUG ] 23:09:34 DEBUG ] 23:09:34 DEBUG } 23:09:34 DEBUG } 23:09:34 DEBUG } 23:09:34 DEBUG } 23:09:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391774" 23:09:34 INFO [http_cmd]: http status code OK 23:09:34 DEBUG --- status code --- 23:09:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:34 DEBUG --- http response --- 23:09:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:34 DEBUG ----- output ----- 23:09:34 DEBUG { 23:09:34 DEBUG "status": "success", 23:09:34 DEBUG "data": { 23:09:34 DEBUG "resultType": "vector", 23:09:34 DEBUG "result": [ 23:09:34 DEBUG { 23:09:34 DEBUG "metric": { 23:09:34 DEBUG "pod": "ds-cts-0" 23:09:34 DEBUG }, 23:09:34 DEBUG "value": [ 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG [...] 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG "pod": "ds-cts-2" 23:09:34 DEBUG }, 23:09:34 DEBUG "value": [ 23:09:34 DEBUG 1713391774, 23:09:34 DEBUG "29.966666666666665" 23:09:34 DEBUG ] 23:09:34 DEBUG } 23:09:34 DEBUG ] 23:09:34 DEBUG } 23:09:34 DEBUG } 23:09:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391774" 23:09:34 INFO [http_cmd]: http status code OK 23:09:34 DEBUG --- status code --- 23:09:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:34 DEBUG --- http response --- 23:09:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:34 DEBUG ----- output ----- 23:09:34 DEBUG { 23:09:34 DEBUG "status": "success", 23:09:34 DEBUG "data": { 23:09:34 DEBUG "resultType": "vector", 23:09:34 DEBUG "result": [ 23:09:34 DEBUG { 23:09:34 DEBUG "metric": { 23:09:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:09:34 DEBUG }, 23:09:34 DEBUG "value": [ 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG [...] 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG 23:09:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:09:34 DEBUG }, 23:09:34 DEBUG "value": [ 23:09:34 DEBUG 1713391774, 23:09:34 DEBUG "1810.1333333333332" 23:09:34 DEBUG ] 23:09:34 DEBUG } 23:09:34 DEBUG ] 23:09:34 DEBUG } 23:09:34 DEBUG } 23:09:35 INFO 23:09:35 INFO [loop_until]: kubectl --namespace=xlou top pods 23:09:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:09:35 INFO [loop_until]: OK (rc = 0) 23:09:35 DEBUG --- stdout --- 23:09:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 12073m 8917Mi am-b59c765b7-bgsxp 10768m 8766Mi am-b59c765b7-ch5q4 11824m 8533Mi ds-cts-0 12753m 7907Mi ds-cts-1 11974m 7757Mi ds-cts-2 12923m 7980Mi ds-idrepo-0 5287m 3221Mi ds-idrepo-1 5268m 2879Mi ds-idrepo-2 5236m 2932Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 967Mi idm-b85f9d978-wnhxb 3m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6729m 2096Mi 23:09:35 DEBUG --- stderr --- 23:09:35 DEBUG 23:09:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391775" 23:09:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391775" 23:09:35 INFO [http_cmd]: http status code OK 23:09:35 DEBUG --- status code --- 23:09:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:35 DEBUG --- http response --- 23:09:35 INFO [http_cmd]: http status code OK 23:09:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:09:35 DEBUG --- status code --- 23:09:35 DEBUG ----- output ----- 23:09:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:35 DEBUG { 23:09:35 DEBUG --- http response --- 23:09:35 DEBUG "status": "success", 23:09:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:35 DEBUG "data": { 23:09:35 DEBUG ----- output ----- 23:09:35 DEBUG "resultType": "vector", 23:09:35 DEBUG { 23:09:35 DEBUG "result": [ 23:09:35 DEBUG "status": "success", 23:09:35 DEBUG { 23:09:35 DEBUG "data": { 23:09:35 DEBUG "metric": { 23:09:35 DEBUG "resultType": "vector", 23:09:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:09:35 DEBUG "result": [ 23:09:35 DEBUG }, 23:09:35 DEBUG { 23:09:35 DEBUG "value": [ 23:09:35 DEBUG "metric": { 23:09:35 DEBUG 23:09:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:09:35 DEBUG }, 23:09:35 DEBUG 23:09:35 DEBUG "value": [ 23:09:35 DEBUG 23:09:35 DEBUG 23:09:35 DEBUG [...] 23:09:35 DEBUG 23:09:35 DEBUG 23:09:35 DEBUG 23:09:35 DEBUG 23:09:35 DEBUG [...] 23:09:35 DEBUG 23:09:35 DEBUG 23:09:35 DEBUG "pod": "am-b59c765b7-7xg4f" 23:09:35 DEBUG 23:09:35 DEBUG }, 23:09:35 DEBUG 23:09:35 DEBUG "value": [ 23:09:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:09:35 DEBUG 1713391775, 23:09:35 DEBUG }, 23:09:35 DEBUG "342403998.68156016" 23:09:35 DEBUG "value": [ 23:09:35 DEBUG ] 23:09:35 DEBUG 1713391775, 23:09:35 DEBUG } 23:09:35 DEBUG "1835.1" 23:09:35 DEBUG ] 23:09:35 DEBUG ] 23:09:35 DEBUG } 23:09:35 DEBUG } 23:09:35 DEBUG } 23:09:35 DEBUG ] 23:09:35 DEBUG } 23:09:35 DEBUG } 23:09:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391775" 23:09:35 INFO [http_cmd]: http status code OK 23:09:35 DEBUG --- status code --- 23:09:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:35 DEBUG --- http response --- 23:09:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:35 DEBUG ----- output ----- 23:09:35 DEBUG { 23:09:35 DEBUG "status": "success", 23:09:35 DEBUG "data": { 23:09:35 DEBUG "resultType": "vector", 23:09:35 DEBUG "result": [ 23:09:35 DEBUG { 23:09:35 DEBUG "metric": { 23:09:35 DEBUG "instance": "10.142.0.107:9100" 23:09:35 DEBUG }, 23:09:35 DEBUG "value": [ 23:09:35 DEBUG 23:09:35 DEBUG 23:09:35 DEBUG 23:09:35 DEBUG [...] 23:09:35 DEBUG 23:09:35 DEBUG 23:09:35 DEBUG 23:09:35 DEBUG "instance": "10.142.0.43:9100" 23:09:35 DEBUG }, 23:09:35 DEBUG "value": [ 23:09:35 DEBUG 1713391775, 23:09:35 DEBUG "229184668672" 23:09:35 DEBUG ] 23:09:35 DEBUG } 23:09:35 DEBUG ] 23:09:35 DEBUG } 23:09:35 DEBUG } 23:09:36 INFO 23:09:36 INFO [loop_until]: kubectl --namespace=xlou top node 23:09:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391776" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391776" 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG --- status code --- 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG --- http response --- 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG { 23:09:36 DEBUG { 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG { 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG { 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "data": { 23:09:36 DEBUG "data": { 23:09:36 DEBUG { 23:09:36 DEBUG "data": { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "data": { 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG "result": [ 23:09:36 DEBUG "result": [ 23:09:36 DEBUG "data": { 23:09:36 DEBUG "result": [ 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG "result": [ 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG { 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG { 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG { 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG { 23:09:36 DEBUG "instance": "10.142.0.107:9100" 23:09:36 DEBUG "result": [ 23:09:36 DEBUG "metric": { 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "instance": "10.142.0.107:9100" 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:09:36 DEBUG "data": { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG }, 23:09:36 DEBUG { 23:09:36 DEBUG "pod": "ds-idrepo-0" 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "ds-cts-0" 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "ds-idrepo-0" 23:09:36 DEBUG "pod": "ds-idrepo-0" 23:09:36 DEBUG "pod": "ds-cts-0" 23:09:36 DEBUG }, 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:09:36 DEBUG }, 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "metric": { 23:09:36 DEBUG }, 23:09:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:09:36 DEBUG }, 23:09:36 DEBUG "pod": "ds-idrepo-0" 23:09:36 DEBUG }, 23:09:36 DEBUG }, 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "pod": "ds-idrepo-0" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "result": [ 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "pod": "ds-idrepo-0" 23:09:36 DEBUG "value": [ 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:09:36 DEBUG }, 23:09:36 DEBUG 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 INFO [loop_until]: OK (rc = 0) 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG --- stdout --- 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5434Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5229m 17% 6154Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11833m 39% 11120Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 2988Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12782m 42% 10166Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 4981Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 12124m 40% 11517Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27088m 90% 2912Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11867m 39% 10616Mi 9% gke-xlou-cdm-ds-0f601751-j488 5660m 18% 5514Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11748m 39% 11347Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5816m 19% 6000Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13060m 43% 9680Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6758m 42% 5867Mi 10% 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG [...] 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG --- stderr --- 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "am-b59c765b7-7xg4f" 23:09:36 DEBUG "instance": "10.142.0.43:9100" 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "instance": "10.142.0.43:9100" 23:09:36 DEBUG 23:09:36 DEBUG "pod": "ds-cts-2" 23:09:36 DEBUG "pod": "am-b59c765b7-7xg4f" 23:09:36 DEBUG "pod": "ds-idrepo-2" 23:09:36 DEBUG }, 23:09:36 DEBUG }, 23:09:36 DEBUG "pod": "ds-idrepo-2" 23:09:36 DEBUG 23:09:36 DEBUG }, 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "ds-cts-2" 23:09:36 DEBUG }, 23:09:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:09:36 DEBUG }, 23:09:36 DEBUG }, 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "value": [ 23:09:36 DEBUG }, 23:09:36 DEBUG "pod": "ds-cts-2" 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:09:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:09:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:09:36 DEBUG "pod": "ds-idrepo-2" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG }, 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG }, 23:09:36 DEBUG }, 23:09:36 DEBUG }, 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG "0" 23:09:36 DEBUG "60945150" 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "1834.3333333333333" 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG "0.12433333333333306" 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG "0.00006666666666666667" 23:09:36 DEBUG "600.0366233290606" 23:09:36 DEBUG "0" 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG "0" 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG ] 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG 1713391776, 23:09:36 DEBUG "0" 23:09:36 DEBUG ] 23:09:36 DEBUG "0.018" 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG "0" 23:09:36 DEBUG } 23:09:36 DEBUG "329277.5" 23:09:36 DEBUG "6.616292377196642" 23:09:36 DEBUG "0" 23:09:36 DEBUG "0" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 DEBUG } 23:10:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391833" 23:10:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391833" 23:10:33 INFO [http_cmd]: http status code OK 23:10:33 DEBUG --- status code --- 23:10:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:33 DEBUG --- http response --- 23:10:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:10:33 ERROR [http_cmd]: ERROR 23:10:33 ERROR --- status code --- 23:10:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:10:33 ERROR --- http response --- 23:10:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:10:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:10:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391834" 23:10:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391834" 23:10:34 INFO [http_cmd]: http status code OK 23:10:34 DEBUG --- status code --- 23:10:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:34 DEBUG --- http response --- 23:10:34 INFO [http_cmd]: http status code OK 23:10:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:34 DEBUG --- status code --- 23:10:34 DEBUG ----- output ----- 23:10:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:34 DEBUG { 23:10:34 DEBUG --- http response --- 23:10:34 DEBUG "status": "success", 23:10:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:10:34 DEBUG "data": { 23:10:34 DEBUG ----- output ----- 23:10:34 DEBUG "resultType": "vector", 23:10:34 DEBUG { 23:10:34 DEBUG "result": [ 23:10:34 DEBUG "status": "success", 23:10:34 DEBUG { 23:10:34 DEBUG "data": { 23:10:34 DEBUG "metric": { 23:10:34 DEBUG "resultType": "vector", 23:10:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:10:34 DEBUG "result": [ 23:10:34 DEBUG }, 23:10:34 DEBUG { 23:10:34 DEBUG "value": [ 23:10:34 DEBUG "metric": { 23:10:34 DEBUG 23:10:34 DEBUG "pod": "ds-idrepo-0" 23:10:34 DEBUG 23:10:34 DEBUG }, 23:10:34 DEBUG 23:10:34 DEBUG "value": [ 23:10:34 DEBUG [...] 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG [...] 23:10:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:10:34 DEBUG 23:10:34 DEBUG }, 23:10:34 DEBUG 23:10:34 DEBUG "value": [ 23:10:34 DEBUG 23:10:34 DEBUG 1713391834, 23:10:34 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 23:10:34 DEBUG "1824.5333333333333" 23:10:34 DEBUG }, 23:10:34 DEBUG ] 23:10:34 DEBUG "value": [ 23:10:34 DEBUG } 23:10:34 DEBUG 1713391834, 23:10:34 DEBUG ] 23:10:34 DEBUG "11.262926872812143" 23:10:34 DEBUG } 23:10:34 DEBUG ] 23:10:34 DEBUG } 23:10:34 DEBUG } 23:10:34 DEBUG ] 23:10:34 DEBUG } 23:10:34 DEBUG } 23:10:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391834" 23:10:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391834" 23:10:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391834" 23:10:34 INFO [http_cmd]: http status code OK 23:10:34 DEBUG --- status code --- 23:10:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:34 DEBUG --- http response --- 23:10:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:34 DEBUG ----- output ----- 23:10:34 DEBUG { 23:10:34 DEBUG "status": "success", 23:10:34 DEBUG "data": { 23:10:34 DEBUG "resultType": "vector", 23:10:34 DEBUG "result": [ 23:10:34 DEBUG { 23:10:34 DEBUG "metric": { 23:10:34 DEBUG "instance": "10.142.0.107:9100" 23:10:34 DEBUG }, 23:10:34 DEBUG "value": [ 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG [...] 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG "instance": "10.142.0.43:9100" 23:10:34 DEBUG }, 23:10:34 DEBUG "value": [ 23:10:34 DEBUG 1713391834, 23:10:34 DEBUG "0.06039999999999945" 23:10:34 DEBUG ] 23:10:34 DEBUG } 23:10:34 DEBUG ] 23:10:34 DEBUG } 23:10:34 DEBUG } 23:10:34 INFO [http_cmd]: http status code OK 23:10:34 DEBUG --- status code --- 23:10:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:34 DEBUG --- http response --- 23:10:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:34 DEBUG ----- output ----- 23:10:34 DEBUG { 23:10:34 DEBUG "status": "success", 23:10:34 DEBUG "data": { 23:10:34 DEBUG "resultType": "vector", 23:10:34 DEBUG "result": [ 23:10:34 DEBUG { 23:10:34 DEBUG "metric": { 23:10:34 DEBUG "instance": "10.142.0.107:9100" 23:10:34 DEBUG }, 23:10:34 DEBUG "value": [ 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG [...] 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG "instance": "10.142.0.43:9100" 23:10:34 DEBUG }, 23:10:34 DEBUG "value": [ 23:10:34 DEBUG 1713391834, 23:10:34 DEBUG "60850718.96666666" 23:10:34 DEBUG ] 23:10:34 DEBUG } 23:10:34 DEBUG ] 23:10:34 DEBUG } 23:10:34 DEBUG } 23:10:34 INFO [http_cmd]: http status code OK 23:10:34 DEBUG --- status code --- 23:10:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:34 DEBUG --- http response --- 23:10:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:34 DEBUG ----- output ----- 23:10:34 DEBUG { 23:10:34 DEBUG "status": "success", 23:10:34 DEBUG "data": { 23:10:34 DEBUG "resultType": "vector", 23:10:34 DEBUG "result": [ 23:10:34 DEBUG { 23:10:34 DEBUG "metric": { 23:10:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:10:34 DEBUG }, 23:10:34 DEBUG "value": [ 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG [...] 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:10:34 DEBUG }, 23:10:34 DEBUG "value": [ 23:10:34 DEBUG 1713391834, 23:10:34 DEBUG "2297626624" 23:10:34 DEBUG ] 23:10:34 DEBUG } 23:10:34 DEBUG ] 23:10:34 DEBUG } 23:10:34 DEBUG } 23:10:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391834" 23:10:34 INFO [http_cmd]: http status code OK 23:10:34 DEBUG --- status code --- 23:10:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:34 DEBUG --- http response --- 23:10:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:34 DEBUG ----- output ----- 23:10:34 DEBUG { 23:10:34 DEBUG "status": "success", 23:10:34 DEBUG "data": { 23:10:34 DEBUG "resultType": "vector", 23:10:34 DEBUG "result": [ 23:10:34 DEBUG { 23:10:34 DEBUG "metric": { 23:10:34 DEBUG "pod": "ds-cts-0" 23:10:34 DEBUG }, 23:10:34 DEBUG "value": [ 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG [...] 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG 23:10:34 DEBUG "pod": "ds-cts-2" 23:10:34 DEBUG }, 23:10:34 DEBUG "value": [ 23:10:34 DEBUG 1713391834, 23:10:34 DEBUG "29.9" 23:10:34 DEBUG ] 23:10:34 DEBUG } 23:10:34 DEBUG ] 23:10:34 DEBUG } 23:10:34 DEBUG } 23:10:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391835" 23:10:35 INFO [http_cmd]: http status code OK 23:10:35 DEBUG --- status code --- 23:10:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:35 DEBUG --- http response --- 23:10:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:35 DEBUG ----- output ----- 23:10:35 DEBUG { 23:10:35 DEBUG "status": "success", 23:10:35 DEBUG "data": { 23:10:35 DEBUG "resultType": "vector", 23:10:35 DEBUG "result": [ 23:10:35 DEBUG { 23:10:35 DEBUG "metric": { 23:10:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:10:35 DEBUG }, 23:10:35 DEBUG "value": [ 23:10:35 DEBUG 23:10:35 DEBUG 23:10:35 DEBUG 23:10:35 DEBUG [...] 23:10:35 DEBUG 23:10:35 DEBUG 23:10:35 DEBUG 23:10:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:10:35 DEBUG }, 23:10:35 DEBUG "value": [ 23:10:35 DEBUG 1713391835, 23:10:35 DEBUG "1835.1666666666667" 23:10:35 DEBUG ] 23:10:35 DEBUG } 23:10:35 DEBUG ] 23:10:35 DEBUG } 23:10:35 DEBUG } 23:10:35 INFO 23:10:35 INFO [loop_until]: kubectl --namespace=xlou top pods 23:10:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:10:35 INFO [loop_until]: OK (rc = 0) 23:10:35 DEBUG --- stdout --- 23:10:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11856m 8917Mi am-b59c765b7-bgsxp 11020m 8767Mi am-b59c765b7-ch5q4 11745m 8535Mi ds-cts-0 12297m 7905Mi ds-cts-1 11847m 7763Mi ds-cts-2 12917m 7983Mi ds-idrepo-0 5420m 3221Mi ds-idrepo-1 5342m 2878Mi ds-idrepo-2 5252m 2933Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6584m 2099Mi 23:10:35 DEBUG --- stderr --- 23:10:35 DEBUG 23:10:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391835" 23:10:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391835" 23:10:35 INFO [http_cmd]: http status code OK 23:10:35 DEBUG --- status code --- 23:10:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:35 DEBUG --- http response --- 23:10:35 INFO [http_cmd]: http status code OK 23:10:35 DEBUG --- status code --- 23:10:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:35 DEBUG ----- output ----- 23:10:35 DEBUG --- http response --- 23:10:35 DEBUG { 23:10:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:10:35 DEBUG "status": "success", 23:10:35 DEBUG ----- output ----- 23:10:35 DEBUG "data": { 23:10:35 DEBUG { 23:10:35 DEBUG "resultType": "vector", 23:10:35 DEBUG "status": "success", 23:10:35 DEBUG "result": [ 23:10:35 DEBUG "data": { 23:10:35 DEBUG { 23:10:35 DEBUG "resultType": "vector", 23:10:35 DEBUG "metric": { 23:10:35 DEBUG "result": [ 23:10:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:10:35 DEBUG { 23:10:35 DEBUG }, 23:10:35 DEBUG "metric": { 23:10:35 DEBUG "value": [ 23:10:35 DEBUG "pod": "ds-idrepo-0" 23:10:35 DEBUG 23:10:35 DEBUG }, 23:10:35 DEBUG 23:10:35 DEBUG "value": [ 23:10:35 DEBUG 23:10:35 DEBUG 23:10:35 DEBUG [...] 23:10:35 DEBUG 23:10:35 DEBUG 23:10:35 DEBUG 23:10:35 DEBUG 23:10:35 DEBUG [...] 23:10:35 DEBUG 23:10:35 DEBUG 23:10:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:10:35 DEBUG 23:10:35 DEBUG }, 23:10:35 DEBUG 23:10:35 DEBUG "value": [ 23:10:35 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 23:10:35 DEBUG 1713391835, 23:10:35 DEBUG }, 23:10:35 DEBUG "1823.9333333333334" 23:10:35 DEBUG "value": [ 23:10:35 DEBUG ] 23:10:35 DEBUG 1713391835, 23:10:35 DEBUG } 23:10:35 DEBUG "127283.50596212952" 23:10:35 DEBUG ] 23:10:35 DEBUG ] 23:10:35 DEBUG } 23:10:35 DEBUG } 23:10:35 DEBUG } 23:10:35 DEBUG ] 23:10:35 DEBUG } 23:10:35 DEBUG } 23:10:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391835" 23:10:35 INFO [http_cmd]: http status code OK 23:10:35 DEBUG --- status code --- 23:10:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:35 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "instance": "10.142.0.107:9100" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "instance": "10.142.0.43:9100" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713391835, 23:10:36 DEBUG "232596313088" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391836" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391836" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391836" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391836" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391836" 23:10:36 INFO 23:10:36 INFO [loop_until]: kubectl --namespace=xlou top node 23:10:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG }, 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG "value": [ 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391836" 23:10:36 DEBUG --- status code --- 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391836" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391836" 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG 23:10:36 DEBUG --- http response --- 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391836" 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391836" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391836" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391836" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391836" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391836" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391836" 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:36 DEBUG --- http response --- 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG [...] 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG ----- output ----- 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG 23:10:36 DEBUG { 23:10:36 DEBUG { 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG --- status code --- 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG { 23:10:36 DEBUG 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG 23:10:36 DEBUG "data": { 23:10:36 DEBUG "data": { 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG { 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG "pod": "ds-idrepo-2" 23:10:36 DEBUG "result": [ 23:10:36 DEBUG "result": [ 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG { 23:10:36 DEBUG "data": { 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG { 23:10:36 DEBUG }, 23:10:36 DEBUG { 23:10:36 DEBUG { 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG "metric": { 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG { 23:10:36 DEBUG { 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "value": [ 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG { 23:10:36 DEBUG "pod": "ds-cts-0" 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:10:36 DEBUG "data": { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG }, 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG "data": { 23:10:36 DEBUG "data": { 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG { 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG "result": [ 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "instance": "10.142.0.107:9100" 23:10:36 DEBUG }, 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "value": [ 23:10:36 DEBUG "data": { 23:10:36 DEBUG { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG "data": { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 DEBUG { 23:10:36 DEBUG "result": [ 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG "result": [ 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG }, 23:10:36 DEBUG "0" 23:10:36 DEBUG "metric": { 23:10:36 DEBUG { 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG { 23:10:36 DEBUG "result": [ 23:10:36 DEBUG 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "data": { 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "result": [ 23:10:36 DEBUG "value": [ 23:10:36 DEBUG ] 23:10:36 DEBUG "instance": "10.142.0.107:9100" 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "metric": { 23:10:36 DEBUG { 23:10:36 DEBUG [...] 23:10:36 DEBUG "result": [ 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 DEBUG { 23:10:36 DEBUG 23:10:36 DEBUG } 23:10:36 DEBUG }, 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:10:36 DEBUG "metric": { 23:10:36 DEBUG 23:10:36 DEBUG { 23:10:36 DEBUG "result": [ 23:10:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:10:36 DEBUG }, 23:10:36 DEBUG "metric": { 23:10:36 DEBUG 23:10:36 DEBUG ] 23:10:36 DEBUG "value": [ 23:10:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:10:36 DEBUG }, 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG }, 23:10:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:10:36 DEBUG 23:10:36 DEBUG "metric": { 23:10:36 DEBUG { 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 DEBUG 23:10:36 DEBUG } 23:10:36 DEBUG 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG "value": [ 23:10:36 DEBUG }, 23:10:36 DEBUG 23:10:36 DEBUG "pod": "ds-cts-0" 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG }, 23:10:36 DEBUG [...] 23:10:36 DEBUG } 23:10:36 DEBUG 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "value": [ 23:10:36 DEBUG "pod": "ds-cts-2" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 DEBUG 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG }, 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG }, 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "instance": "10.142.0.43:9100" 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "value": [ 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG }, 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "instance": "10.142.0.43:9100" 23:10:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:10:36 DEBUG 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG [...] 23:10:36 DEBUG "0" 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG [...] 23:10:36 DEBUG }, 23:10:36 DEBUG }, 23:10:36 DEBUG 23:10:36 DEBUG "60849869.73333333" 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG ] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "ds-idrepo-2" 23:10:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "value": [ 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG ] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG } 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG }, 23:10:36 DEBUG }, 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:10:36 DEBUG } 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG ] 23:10:36 DEBUG 23:10:36 DEBUG "pod": "ds-idrepo-2" 23:10:36 DEBUG 23:10:36 DEBUG "value": [ 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG "pod": "ds-cts-2" 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG "329238" 23:10:36 DEBUG }, 23:10:36 DEBUG ] 23:10:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:10:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:10:36 DEBUG } 23:10:36 DEBUG "pod": "ds-cts-2" 23:10:36 DEBUG }, 23:10:36 DEBUG 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG "pod": "ds-idrepo-2" 23:10:36 DEBUG }, 23:10:36 DEBUG "0.04366666666666675" 23:10:36 DEBUG ] 23:10:36 DEBUG "value": [ 23:10:36 DEBUG } 23:10:36 DEBUG }, 23:10:36 DEBUG }, 23:10:36 DEBUG } 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG "0" 23:10:36 DEBUG "0" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG } 23:10:36 DEBUG "value": [ 23:10:36 DEBUG "value": [ 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG "pod": "ds-idrepo-2" 23:10:36 DEBUG ] 23:10:36 DEBUG ] 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG "1824.7333333333333" 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG 1713391836, 23:10:36 INFO [loop_until]: OK (rc = 0) 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG "2721.6802248188137" 23:10:36 DEBUG }, 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG "0" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG "0.018" 23:10:36 DEBUG "6.604361621096271" 23:10:36 DEBUG --- stdout --- 23:10:36 DEBUG "0.00003333333333333336" 23:10:36 DEBUG ] 23:10:36 DEBUG "value": [ 23:10:36 DEBUG ] 23:10:36 DEBUG ] 23:10:36 DEBUG "0" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG ] 23:10:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 5455Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5218m 17% 6150Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11749m 39% 11124Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3002Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12436m 41% 10183Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 4982Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11888m 39% 11530Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27751m 92% 2928Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11852m 39% 10616Mi 9% gke-xlou-cdm-ds-0f601751-j488 5634m 18% 5522Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11739m 39% 11343Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5759m 19% 6003Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 12953m 43% 9695Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6515m 41% 5869Mi 10% 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG 1713391836, 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG --- stderr --- 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG "0" 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG ] 23:10:36 DEBUG ] 23:10:36 DEBUG 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG } 23:11:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391893" 23:11:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391893" 23:11:33 INFO [http_cmd]: http status code OK 23:11:33 DEBUG --- status code --- 23:11:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:33 DEBUG --- http response --- 23:11:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:11:33 ERROR [http_cmd]: ERROR 23:11:33 ERROR --- status code --- 23:11:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:11:33 ERROR --- http response --- 23:11:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:11:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:11:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391894" 23:11:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391894" 23:11:34 INFO [http_cmd]: http status code OK 23:11:34 INFO [http_cmd]: http status code OK 23:11:34 DEBUG --- status code --- 23:11:34 DEBUG --- status code --- 23:11:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:34 DEBUG --- http response --- 23:11:34 DEBUG --- http response --- 23:11:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:11:34 DEBUG ----- output ----- 23:11:34 DEBUG ----- output ----- 23:11:34 DEBUG { 23:11:34 DEBUG { 23:11:34 DEBUG "status": "success", 23:11:34 DEBUG "status": "success", 23:11:34 DEBUG "data": { 23:11:34 DEBUG "data": { 23:11:34 DEBUG "resultType": "vector", 23:11:34 DEBUG "resultType": "vector", 23:11:34 DEBUG "result": [ 23:11:34 DEBUG "result": [ 23:11:34 DEBUG { 23:11:34 DEBUG { 23:11:34 DEBUG "metric": { 23:11:34 DEBUG "metric": { 23:11:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:11:34 DEBUG "pod": "ds-idrepo-0" 23:11:34 DEBUG }, 23:11:34 DEBUG }, 23:11:34 DEBUG "value": [ 23:11:34 DEBUG "value": [ 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG [...] 23:11:34 DEBUG [...] 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:11:34 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:11:34 DEBUG }, 23:11:34 DEBUG }, 23:11:34 DEBUG "value": [ 23:11:34 DEBUG "value": [ 23:11:34 DEBUG 1713391894, 23:11:34 DEBUG 1713391894, 23:11:34 DEBUG "1821.6333333333332" 23:11:34 DEBUG "0.003710939205839009" 23:11:34 DEBUG ] 23:11:34 DEBUG ] 23:11:34 DEBUG } 23:11:34 DEBUG } 23:11:34 DEBUG ] 23:11:34 DEBUG ] 23:11:34 DEBUG } 23:11:34 DEBUG } 23:11:34 DEBUG } 23:11:34 DEBUG } 23:11:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391894" 23:11:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391894" 23:11:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391894" 23:11:34 INFO [http_cmd]: http status code OK 23:11:34 DEBUG --- status code --- 23:11:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:34 DEBUG --- http response --- 23:11:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:34 DEBUG ----- output ----- 23:11:34 DEBUG { 23:11:34 DEBUG "status": "success", 23:11:34 DEBUG "data": { 23:11:34 DEBUG "resultType": "vector", 23:11:34 DEBUG "result": [ 23:11:34 DEBUG { 23:11:34 INFO [http_cmd]: http status code OK 23:11:34 DEBUG "metric": { 23:11:34 DEBUG --- status code --- 23:11:34 DEBUG "instance": "10.142.0.107:9100" 23:11:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:34 DEBUG }, 23:11:34 DEBUG --- http response --- 23:11:34 DEBUG "value": [ 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG [...] 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG "instance": "10.142.0.43:9100" 23:11:34 DEBUG }, 23:11:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:34 DEBUG "value": [ 23:11:34 INFO [http_cmd]: http status code OK 23:11:34 DEBUG ----- output ----- 23:11:34 DEBUG 1713391894, 23:11:34 DEBUG --- status code --- 23:11:34 DEBUG { 23:11:34 DEBUG "0.13219999999999932" 23:11:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:34 DEBUG "status": "success", 23:11:34 DEBUG ] 23:11:34 DEBUG --- http response --- 23:11:34 DEBUG "data": { 23:11:34 DEBUG } 23:11:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:34 DEBUG "resultType": "vector", 23:11:34 DEBUG ] 23:11:34 DEBUG ----- output ----- 23:11:34 DEBUG "result": [ 23:11:34 DEBUG } 23:11:34 DEBUG { 23:11:34 DEBUG { 23:11:34 DEBUG } 23:11:34 DEBUG "status": "success", 23:11:34 DEBUG "metric": { 23:11:34 DEBUG "data": { 23:11:34 DEBUG "instance": "10.142.0.107:9100" 23:11:34 DEBUG "resultType": "vector", 23:11:34 DEBUG }, 23:11:34 DEBUG "result": [ 23:11:34 DEBUG "value": [ 23:11:34 DEBUG { 23:11:34 DEBUG 23:11:34 DEBUG "metric": { 23:11:34 DEBUG 23:11:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:11:34 DEBUG 23:11:34 DEBUG }, 23:11:34 DEBUG [...] 23:11:34 DEBUG "value": [ 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG "instance": "10.142.0.43:9100" 23:11:34 DEBUG [...] 23:11:34 DEBUG }, 23:11:34 DEBUG 23:11:34 DEBUG "value": [ 23:11:34 DEBUG 23:11:34 DEBUG 1713391894, 23:11:34 DEBUG 23:11:34 DEBUG "60978782.03333333" 23:11:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:11:34 DEBUG ] 23:11:34 DEBUG }, 23:11:34 DEBUG } 23:11:34 DEBUG "value": [ 23:11:34 DEBUG ] 23:11:34 DEBUG 1713391894, 23:11:34 DEBUG } 23:11:34 DEBUG "2300735488" 23:11:34 DEBUG } 23:11:34 DEBUG ] 23:11:34 DEBUG } 23:11:34 DEBUG ] 23:11:34 DEBUG } 23:11:34 DEBUG } 23:11:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391894" 23:11:34 INFO [http_cmd]: http status code OK 23:11:34 DEBUG --- status code --- 23:11:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:34 DEBUG --- http response --- 23:11:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:34 DEBUG ----- output ----- 23:11:34 DEBUG { 23:11:34 DEBUG "status": "success", 23:11:34 DEBUG "data": { 23:11:34 DEBUG "resultType": "vector", 23:11:34 DEBUG "result": [ 23:11:34 DEBUG { 23:11:34 DEBUG "metric": { 23:11:34 DEBUG "pod": "ds-cts-0" 23:11:34 DEBUG }, 23:11:34 DEBUG "value": [ 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG [...] 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG 23:11:34 DEBUG "pod": "ds-cts-2" 23:11:34 DEBUG }, 23:11:34 DEBUG "value": [ 23:11:34 DEBUG 1713391894, 23:11:34 DEBUG "29.933333333333334" 23:11:34 DEBUG ] 23:11:34 DEBUG } 23:11:34 DEBUG ] 23:11:34 DEBUG } 23:11:34 DEBUG } 23:11:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391895" 23:11:35 INFO [http_cmd]: http status code OK 23:11:35 DEBUG --- status code --- 23:11:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:35 DEBUG --- http response --- 23:11:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:35 DEBUG ----- output ----- 23:11:35 DEBUG { 23:11:35 DEBUG "status": "success", 23:11:35 DEBUG "data": { 23:11:35 DEBUG "resultType": "vector", 23:11:35 DEBUG "result": [ 23:11:35 DEBUG { 23:11:35 DEBUG "metric": { 23:11:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:11:35 DEBUG }, 23:11:35 DEBUG "value": [ 23:11:35 DEBUG 23:11:35 DEBUG 23:11:35 DEBUG 23:11:35 DEBUG [...] 23:11:35 DEBUG 23:11:35 DEBUG 23:11:35 DEBUG 23:11:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:11:35 DEBUG }, 23:11:35 DEBUG "value": [ 23:11:35 DEBUG 1713391895, 23:11:35 DEBUG "1827.1" 23:11:35 DEBUG ] 23:11:35 DEBUG } 23:11:35 DEBUG ] 23:11:35 DEBUG } 23:11:35 DEBUG } 23:11:35 INFO 23:11:35 INFO [loop_until]: kubectl --namespace=xlou top pods 23:11:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:11:35 INFO [loop_until]: OK (rc = 0) 23:11:35 DEBUG --- stdout --- 23:11:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11911m 8915Mi am-b59c765b7-bgsxp 11136m 8769Mi am-b59c765b7-ch5q4 12149m 8536Mi ds-cts-0 12523m 7923Mi ds-cts-1 11610m 7771Mi ds-cts-2 13555m 7981Mi ds-idrepo-0 5208m 3221Mi ds-idrepo-1 5358m 2879Mi ds-idrepo-2 5270m 2932Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6594m 2100Mi 23:11:35 DEBUG --- stderr --- 23:11:35 DEBUG 23:11:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391895" 23:11:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391895" 23:11:35 INFO [http_cmd]: http status code OK 23:11:35 INFO [http_cmd]: http status code OK 23:11:35 DEBUG --- status code --- 23:11:35 DEBUG --- status code --- 23:11:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:35 DEBUG --- http response --- 23:11:35 DEBUG --- http response --- 23:11:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:11:35 DEBUG ----- output ----- 23:11:35 DEBUG ----- output ----- 23:11:35 DEBUG { 23:11:35 DEBUG { 23:11:35 DEBUG "status": "success", 23:11:35 DEBUG "status": "success", 23:11:35 DEBUG "data": { 23:11:35 DEBUG "data": { 23:11:35 DEBUG "resultType": "vector", 23:11:35 DEBUG "resultType": "vector", 23:11:35 DEBUG "result": [ 23:11:35 DEBUG "result": [ 23:11:35 DEBUG { 23:11:35 DEBUG { 23:11:35 DEBUG "metric": { 23:11:35 DEBUG "metric": { 23:11:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:11:35 DEBUG "pod": "ds-idrepo-0" 23:11:35 DEBUG }, 23:11:35 DEBUG }, 23:11:35 DEBUG "value": [ 23:11:35 DEBUG "value": [ 23:11:35 DEBUG 23:11:35 DEBUG 23:11:35 DEBUG 23:11:35 DEBUG 23:11:35 DEBUG 23:11:35 DEBUG [...] 23:11:35 DEBUG 23:11:35 DEBUG 23:11:35 DEBUG [...] 23:11:35 DEBUG 23:11:35 DEBUG 23:11:35 DEBUG 23:11:35 DEBUG 23:11:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:11:35 DEBUG 23:11:35 DEBUG }, 23:11:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:11:35 DEBUG "value": [ 23:11:35 DEBUG }, 23:11:35 DEBUG 1713391895, 23:11:35 DEBUG "1821.9333333333334" 23:11:35 DEBUG "value": [ 23:11:35 DEBUG ] 23:11:35 DEBUG 1713391895, 23:11:35 DEBUG } 23:11:35 DEBUG "12265.385695124616" 23:11:35 DEBUG ] 23:11:35 DEBUG ] 23:11:35 DEBUG } 23:11:35 DEBUG } 23:11:35 DEBUG } 23:11:35 DEBUG ] 23:11:35 DEBUG } 23:11:35 DEBUG } 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391896" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "instance": "10.142.0.107:9100" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "instance": "10.142.0.43:9100" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG "235279694848" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391896" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391896" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391896" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391896" 23:11:36 DEBUG --- status code --- 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391896" 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391896" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391896" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391896" 23:11:36 DEBUG --- status code --- 23:11:36 INFO 23:11:36 DEBUG --- http response --- 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391896" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391896" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391896" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391896" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391896" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391896" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391896" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 INFO [loop_until]: kubectl --namespace=xlou top node 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- http response --- 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG --- status code --- 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG { 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- status code --- 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG "data": { 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG "data": { 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "data": { 23:11:36 DEBUG "data": { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "data": { 23:11:36 DEBUG { 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG { 23:11:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "data": { 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG }, 23:11:36 DEBUG "result": [ 23:11:36 DEBUG "result": [ 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG "result": [ 23:11:36 DEBUG "data": { 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG }, 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG "data": { 23:11:36 DEBUG "value": [ 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG "result": [ 23:11:36 DEBUG "result": [ 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "value": [ 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "result": [ 23:11:36 DEBUG 23:11:36 DEBUG "instance": "10.142.0.107:9100" 23:11:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:11:36 DEBUG "pod": "ds-cts-0" 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "instance": "10.142.0.107:9100" 23:11:36 DEBUG { 23:11:36 DEBUG 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG { 23:11:36 DEBUG 23:11:36 DEBUG }, 23:11:36 DEBUG }, 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:11:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG }, 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG }, 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG [...] 23:11:36 DEBUG "metric": { 23:11:36 DEBUG [...] 23:11:36 DEBUG "value": [ 23:11:36 DEBUG "value": [ 23:11:36 DEBUG }, 23:11:36 DEBUG }, 23:11:36 DEBUG }, 23:11:36 DEBUG }, 23:11:36 DEBUG 23:11:36 DEBUG "value": [ 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG }, 23:11:36 DEBUG }, 23:11:36 DEBUG 23:11:36 DEBUG "pod": "ds-cts-0" 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "value": [ 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG "value": [ 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG }, 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "idm-b85f9d978-txpwv" 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG }, 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG }, 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG [...] 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG "1821.6666666666667" 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG }, 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "ds-cts-2" 23:11:36 DEBUG "0" 23:11:36 DEBUG 23:11:36 DEBUG ] 23:11:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:11:36 DEBUG 23:11:36 DEBUG "pod": "ds-idrepo-2" 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG "instance": "10.142.0.43:9100" 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG }, 23:11:36 DEBUG ] 23:11:36 DEBUG "instance": "10.142.0.43:9100" 23:11:36 DEBUG } 23:11:36 DEBUG }, 23:11:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:11:36 DEBUG }, 23:11:36 DEBUG 23:11:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:11:36 DEBUG "pod": "ds-cts-2" 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG "pod": "ds-idrepo-2" 23:11:36 DEBUG }, 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "value": [ 23:11:36 DEBUG } 23:11:36 DEBUG }, 23:11:36 DEBUG ] 23:11:36 DEBUG "value": [ 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG "pod": "ds-idrepo-2" 23:11:36 DEBUG }, 23:11:36 DEBUG }, 23:11:36 DEBUG "0.018" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG "pod": "idm-b85f9d978-txpwv" 23:11:36 DEBUG 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG ] 23:11:36 DEBUG "value": [ 23:11:36 DEBUG } 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG "value": [ 23:11:36 DEBUG ] 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG }, 23:11:36 DEBUG "pod": "ds-cts-2" 23:11:36 DEBUG "0" 23:11:36 DEBUG } 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG } 23:11:36 DEBUG "328508.5" 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG "0" 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG } 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG "0.06166666666666627" 23:11:36 DEBUG "value": [ 23:11:36 DEBUG }, 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG "60977943.13333333" 23:11:36 DEBUG ] 23:11:36 DEBUG "6.5986040251600455" 23:11:36 DEBUG ] 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG "0" 23:11:36 DEBUG "0" 23:11:36 DEBUG ] 23:11:36 DEBUG "0" 23:11:36 DEBUG ] 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG "0" 23:11:36 DEBUG ] 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG "428.6575055141704" 23:11:36 DEBUG 1713391896, 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG ] 23:11:36 DEBUG "0.00003333333333333331" 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG ] 23:11:36 DEBUG ] 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [loop_until]: OK (rc = 0) 23:11:36 DEBUG --- stdout --- 23:11:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5453Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5230m 17% 6155Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11951m 40% 11119Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 2986Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12355m 41% 10174Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 136m 0% 4987Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11927m 39% 11536Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27928m 93% 2923Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11913m 39% 10614Mi 9% gke-xlou-cdm-ds-0f601751-j488 5469m 18% 5513Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11516m 38% 11335Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5813m 19% 5988Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13043m 43% 9695Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6857m 43% 5870Mi 10% 23:11:36 DEBUG --- stderr --- 23:11:36 DEBUG 23:12:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391953" 23:12:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713391953" 23:12:33 INFO [http_cmd]: http status code OK 23:12:33 DEBUG --- status code --- 23:12:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:33 DEBUG --- http response --- 23:12:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:12:33 ERROR [http_cmd]: ERROR 23:12:33 ERROR --- status code --- 23:12:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:12:33 ERROR --- http response --- 23:12:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:12:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:12:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391954" 23:12:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391954" 23:12:34 INFO [http_cmd]: http status code OK 23:12:34 DEBUG --- status code --- 23:12:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:34 DEBUG --- http response --- 23:12:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:12:34 DEBUG ----- output ----- 23:12:34 DEBUG { 23:12:34 DEBUG "status": "success", 23:12:34 DEBUG "data": { 23:12:34 DEBUG "resultType": "vector", 23:12:34 DEBUG "result": [ 23:12:34 DEBUG { 23:12:34 DEBUG "metric": { 23:12:34 DEBUG "pod": "ds-idrepo-1" 23:12:34 DEBUG }, 23:12:34 DEBUG "value": [ 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG [...] 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG "pod": "ds-idrepo-2" 23:12:34 DEBUG }, 23:12:34 DEBUG "value": [ 23:12:34 DEBUG 1713391954, 23:12:34 DEBUG "5.274481821871701" 23:12:34 DEBUG ] 23:12:34 DEBUG } 23:12:34 DEBUG ] 23:12:34 DEBUG } 23:12:34 DEBUG } 23:12:34 INFO [http_cmd]: http status code OK 23:12:34 DEBUG --- status code --- 23:12:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:34 DEBUG --- http response --- 23:12:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:34 DEBUG ----- output ----- 23:12:34 DEBUG { 23:12:34 DEBUG "status": "success", 23:12:34 DEBUG "data": { 23:12:34 DEBUG "resultType": "vector", 23:12:34 DEBUG "result": [ 23:12:34 DEBUG { 23:12:34 DEBUG "metric": { 23:12:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:12:34 DEBUG }, 23:12:34 DEBUG "value": [ 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG [...] 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:12:34 DEBUG }, 23:12:34 DEBUG "value": [ 23:12:34 DEBUG 1713391954, 23:12:34 DEBUG "1829.8666666666666" 23:12:34 DEBUG ] 23:12:34 DEBUG } 23:12:34 DEBUG ] 23:12:34 DEBUG } 23:12:34 DEBUG } 23:12:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391954" 23:12:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713391954" 23:12:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391954" 23:12:34 INFO [http_cmd]: http status code OK 23:12:34 DEBUG --- status code --- 23:12:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:34 DEBUG --- http response --- 23:12:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:34 DEBUG ----- output ----- 23:12:34 DEBUG { 23:12:34 DEBUG "status": "success", 23:12:34 DEBUG "data": { 23:12:34 DEBUG "resultType": "vector", 23:12:34 DEBUG "result": [ 23:12:34 DEBUG { 23:12:34 DEBUG "metric": { 23:12:34 DEBUG "instance": "10.142.0.107:9100" 23:12:34 DEBUG }, 23:12:34 DEBUG "value": [ 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG [...] 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG "instance": "10.142.0.43:9100" 23:12:34 DEBUG }, 23:12:34 DEBUG "value": [ 23:12:34 DEBUG 1713391954, 23:12:34 DEBUG "0.04660000000000082" 23:12:34 DEBUG ] 23:12:34 DEBUG } 23:12:34 DEBUG ] 23:12:34 DEBUG } 23:12:34 DEBUG } 23:12:34 INFO [http_cmd]: http status code OK 23:12:34 DEBUG --- status code --- 23:12:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:34 DEBUG --- http response --- 23:12:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:34 DEBUG ----- output ----- 23:12:34 DEBUG { 23:12:34 DEBUG "status": "success", 23:12:34 DEBUG "data": { 23:12:34 DEBUG "resultType": "vector", 23:12:34 DEBUG "result": [ 23:12:34 DEBUG { 23:12:34 DEBUG "metric": { 23:12:34 DEBUG "instance": "10.142.0.107:9100" 23:12:34 DEBUG }, 23:12:34 DEBUG "value": [ 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG [...] 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG "instance": "10.142.0.43:9100" 23:12:34 DEBUG }, 23:12:34 DEBUG "value": [ 23:12:34 DEBUG 1713391954, 23:12:34 DEBUG "60957612.23333333" 23:12:34 DEBUG ] 23:12:34 DEBUG } 23:12:34 DEBUG ] 23:12:34 DEBUG } 23:12:34 DEBUG } 23:12:34 INFO [http_cmd]: http status code OK 23:12:34 DEBUG --- status code --- 23:12:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:34 DEBUG --- http response --- 23:12:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:34 DEBUG ----- output ----- 23:12:34 DEBUG { 23:12:34 DEBUG "status": "success", 23:12:34 DEBUG "data": { 23:12:34 DEBUG "resultType": "vector", 23:12:34 DEBUG "result": [ 23:12:34 DEBUG { 23:12:34 DEBUG "metric": { 23:12:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:12:34 DEBUG }, 23:12:34 DEBUG "value": [ 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG [...] 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG 23:12:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:12:34 DEBUG }, 23:12:34 DEBUG "value": [ 23:12:34 DEBUG 1713391954, 23:12:34 DEBUG "2303422464" 23:12:34 DEBUG ] 23:12:34 DEBUG } 23:12:34 DEBUG ] 23:12:34 DEBUG } 23:12:34 DEBUG } 23:12:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391955" 23:12:35 INFO [http_cmd]: http status code OK 23:12:35 DEBUG --- status code --- 23:12:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:35 DEBUG --- http response --- 23:12:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:35 DEBUG ----- output ----- 23:12:35 DEBUG { 23:12:35 DEBUG "status": "success", 23:12:35 DEBUG "data": { 23:12:35 DEBUG "resultType": "vector", 23:12:35 DEBUG "result": [ 23:12:35 DEBUG { 23:12:35 DEBUG "metric": { 23:12:35 DEBUG "pod": "ds-cts-0" 23:12:35 DEBUG }, 23:12:35 DEBUG "value": [ 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG [...] 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG "pod": "ds-cts-2" 23:12:35 DEBUG }, 23:12:35 DEBUG "value": [ 23:12:35 DEBUG 1713391955, 23:12:35 DEBUG "29.966666666666665" 23:12:35 DEBUG ] 23:12:35 DEBUG } 23:12:35 DEBUG ] 23:12:35 DEBUG } 23:12:35 DEBUG } 23:12:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391955" 23:12:35 INFO [http_cmd]: http status code OK 23:12:35 DEBUG --- status code --- 23:12:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:35 DEBUG --- http response --- 23:12:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:35 DEBUG ----- output ----- 23:12:35 DEBUG { 23:12:35 DEBUG "status": "success", 23:12:35 DEBUG "data": { 23:12:35 DEBUG "resultType": "vector", 23:12:35 DEBUG "result": [ 23:12:35 DEBUG { 23:12:35 DEBUG "metric": { 23:12:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:12:35 DEBUG }, 23:12:35 DEBUG "value": [ 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG [...] 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:12:35 DEBUG }, 23:12:35 DEBUG "value": [ 23:12:35 DEBUG 1713391955, 23:12:35 DEBUG "1834.3" 23:12:35 DEBUG ] 23:12:35 DEBUG } 23:12:35 DEBUG ] 23:12:35 DEBUG } 23:12:35 DEBUG } 23:12:35 INFO 23:12:35 INFO [loop_until]: kubectl --namespace=xlou top pods 23:12:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:12:35 INFO [loop_until]: OK (rc = 0) 23:12:35 DEBUG --- stdout --- 23:12:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11687m 8914Mi am-b59c765b7-bgsxp 11430m 8770Mi am-b59c765b7-ch5q4 12264m 8534Mi ds-cts-0 12434m 7919Mi ds-cts-1 11509m 7789Mi ds-cts-2 13062m 7997Mi ds-idrepo-0 5368m 3221Mi ds-idrepo-1 5377m 2879Mi ds-idrepo-2 5355m 2932Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 3m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6572m 2101Mi 23:12:35 DEBUG --- stderr --- 23:12:35 DEBUG 23:12:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391955" 23:12:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391955" 23:12:35 INFO [http_cmd]: http status code OK 23:12:35 INFO [http_cmd]: http status code OK 23:12:35 DEBUG --- status code --- 23:12:35 DEBUG --- status code --- 23:12:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:35 DEBUG --- http response --- 23:12:35 DEBUG --- http response --- 23:12:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:35 DEBUG ----- output ----- 23:12:35 DEBUG { 23:12:35 DEBUG "status": "success", 23:12:35 DEBUG "data": { 23:12:35 DEBUG "resultType": "vector", 23:12:35 DEBUG "result": [ 23:12:35 DEBUG { 23:12:35 DEBUG "metric": { 23:12:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:12:35 DEBUG }, 23:12:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:12:35 DEBUG "value": [ 23:12:35 DEBUG ----- output ----- 23:12:35 DEBUG 23:12:35 DEBUG { 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG [...] 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:12:35 DEBUG }, 23:12:35 DEBUG "value": [ 23:12:35 DEBUG "status": "success", 23:12:35 DEBUG 1713391955, 23:12:35 DEBUG "data": { 23:12:35 DEBUG "1829.2333333333333" 23:12:35 DEBUG "resultType": "vector", 23:12:35 DEBUG ] 23:12:35 DEBUG "result": [ 23:12:35 DEBUG } 23:12:35 DEBUG { 23:12:35 DEBUG ] 23:12:35 DEBUG "metric": { 23:12:35 DEBUG } 23:12:35 DEBUG "pod": "ds-idrepo-0" 23:12:35 DEBUG } 23:12:35 DEBUG }, 23:12:35 DEBUG "value": [ 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG [...] 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG 23:12:35 DEBUG "pod": "am-b59c765b7-7xg4f" 23:12:35 DEBUG }, 23:12:35 DEBUG "value": [ 23:12:35 DEBUG 1713391955, 23:12:35 DEBUG "19223.344145694413" 23:12:35 DEBUG ] 23:12:35 DEBUG } 23:12:35 DEBUG ] 23:12:35 DEBUG } 23:12:35 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391956" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "instance": "10.142.0.107:9100" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "instance": "10.142.0.43:9100" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG "237766456320" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391956" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713391956" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG --- status code --- 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 INFO 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- status code --- 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 INFO [loop_until]: kubectl --namespace=xlou top node 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG { 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG "data": { 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG { 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG { 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG { 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG { 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "data": { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "result": [ 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "data": { 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "data": { 23:12:36 DEBUG "data": { 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG { 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG "result": [ 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG }, 23:12:36 DEBUG "result": [ 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG { 23:12:36 DEBUG "result": [ 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG "result": [ 23:12:36 DEBUG "result": [ 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "value": [ 23:12:36 DEBUG { 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG { 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG { 23:12:36 DEBUG { 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG }, 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG "metric": { 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "ds-cts-0" 23:12:36 DEBUG }, 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:12:36 DEBUG 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:12:36 DEBUG }, 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:12:36 DEBUG "pod": "ds-cts-0" 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "instance": "10.142.0.107:9100" 23:12:36 DEBUG }, 23:12:36 DEBUG 23:12:36 DEBUG }, 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "instance": "10.142.0.107:9100" 23:12:36 DEBUG 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG }, 23:12:36 DEBUG }, 23:12:36 DEBUG }, 23:12:36 DEBUG }, 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG [...] 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG }, 23:12:36 DEBUG 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:12:36 DEBUG [...] 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG [...] 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG }, 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "am-b59c765b7-7xg4f" 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "ds-cts-2" 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:12:36 DEBUG 23:12:36 DEBUG }, 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG }, 23:12:36 DEBUG "pod": "ds-idrepo-2" 23:12:36 DEBUG "pod": "ds-cts-2" 23:12:36 DEBUG 23:12:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:12:36 DEBUG "0.018" 23:12:36 DEBUG "pod": "ds-idrepo-2" 23:12:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:12:36 DEBUG }, 23:12:36 DEBUG 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "pod": "am-b59c765b7-7xg4f" 23:12:36 DEBUG 23:12:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:12:36 DEBUG "pod": "ds-idrepo-2" 23:12:36 DEBUG "value": [ 23:12:36 DEBUG }, 23:12:36 DEBUG }, 23:12:36 DEBUG "instance": "10.142.0.43:9100" 23:12:36 DEBUG }, 23:12:36 DEBUG ] 23:12:36 DEBUG }, 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "instance": "10.142.0.43:9100" 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG }, 23:12:36 DEBUG "pod": "ds-cts-2" 23:12:36 DEBUG }, 23:12:36 DEBUG }, 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "value": [ 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG } 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG }, 23:12:36 DEBUG "0" 23:12:36 DEBUG "value": [ 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "0" 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG ] 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG "328196.5" 23:12:36 DEBUG "value": [ 23:12:36 DEBUG ] 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG ] 23:12:36 DEBUG "0" 23:12:36 DEBUG "0" 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG "1829.8333333333333" 23:12:36 DEBUG } 23:12:36 DEBUG "0" 23:12:36 DEBUG "6.627378988494238" 23:12:36 DEBUG ] 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG } 23:12:36 DEBUG "410.39294471929446" 23:12:36 DEBUG 1713391956, 23:12:36 DEBUG "0" 23:12:36 DEBUG "0" 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG ] 23:12:36 DEBUG "0.03200000000000051" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG "60956681.43333333" 23:12:36 DEBUG ] 23:12:36 DEBUG ] 23:12:36 DEBUG "0" 23:12:36 DEBUG ] 23:12:36 DEBUG ] 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [loop_until]: OK (rc = 0) 23:12:36 DEBUG --- stdout --- 23:12:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 5452Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5242m 17% 6149Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11937m 39% 11120Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 2982Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12513m 41% 10181Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 4965Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11963m 40% 11521Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28057m 93% 2923Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11473m 38% 10608Mi 9% gke-xlou-cdm-ds-0f601751-j488 5677m 19% 5500Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11344m 37% 11350Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5791m 19% 5993Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13053m 43% 9702Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6796m 42% 5867Mi 10% 23:12:36 DEBUG --- stderr --- 23:12:36 DEBUG 23:13:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392013" 23:13:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713392013" 23:13:33 INFO [http_cmd]: http status code OK 23:13:33 DEBUG --- status code --- 23:13:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:33 DEBUG --- http response --- 23:13:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:13:33 ERROR [http_cmd]: ERROR 23:13:33 ERROR --- status code --- 23:13:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:13:33 ERROR --- http response --- 23:13:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:13:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:13:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392014" 23:13:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392014" 23:13:34 INFO [http_cmd]: http status code OK 23:13:34 DEBUG --- status code --- 23:13:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:34 DEBUG --- http response --- 23:13:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:13:34 DEBUG ----- output ----- 23:13:34 DEBUG { 23:13:34 DEBUG "status": "success", 23:13:34 DEBUG "data": { 23:13:34 DEBUG "resultType": "vector", 23:13:34 DEBUG "result": [ 23:13:34 DEBUG { 23:13:34 DEBUG "metric": { 23:13:34 DEBUG "pod": "ds-idrepo-0" 23:13:34 DEBUG }, 23:13:34 DEBUG "value": [ 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG [...] 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG "pod": "am-b59c765b7-7xg4f" 23:13:34 DEBUG }, 23:13:34 DEBUG "value": [ 23:13:34 DEBUG 1713392014, 23:13:34 DEBUG "0" 23:13:34 DEBUG ] 23:13:34 DEBUG } 23:13:34 DEBUG ] 23:13:34 DEBUG } 23:13:34 DEBUG } 23:13:34 INFO [http_cmd]: http status code OK 23:13:34 DEBUG --- status code --- 23:13:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:34 DEBUG --- http response --- 23:13:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:34 DEBUG ----- output ----- 23:13:34 DEBUG { 23:13:34 DEBUG "status": "success", 23:13:34 DEBUG "data": { 23:13:34 DEBUG "resultType": "vector", 23:13:34 DEBUG "result": [ 23:13:34 DEBUG { 23:13:34 DEBUG "metric": { 23:13:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:13:34 DEBUG }, 23:13:34 DEBUG "value": [ 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG [...] 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:13:34 DEBUG }, 23:13:34 DEBUG "value": [ 23:13:34 DEBUG 1713392014, 23:13:34 DEBUG "1814.5333333333333" 23:13:34 DEBUG ] 23:13:34 DEBUG } 23:13:34 DEBUG ] 23:13:34 DEBUG } 23:13:34 DEBUG } 23:13:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392014" 23:13:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713392014" 23:13:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392014" 23:13:34 INFO [http_cmd]: http status code OK 23:13:34 DEBUG --- status code --- 23:13:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:34 DEBUG --- http response --- 23:13:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:34 DEBUG ----- output ----- 23:13:34 DEBUG { 23:13:34 DEBUG "status": "success", 23:13:34 DEBUG "data": { 23:13:34 DEBUG "resultType": "vector", 23:13:34 DEBUG "result": [ 23:13:34 DEBUG { 23:13:34 DEBUG "metric": { 23:13:34 DEBUG "instance": "10.142.0.107:9100" 23:13:34 DEBUG }, 23:13:34 DEBUG "value": [ 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG [...] 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG "instance": "10.142.0.43:9100" 23:13:34 DEBUG }, 23:13:34 DEBUG "value": [ 23:13:34 DEBUG 1713392014, 23:13:34 DEBUG "0.15683333333333377" 23:13:34 DEBUG ] 23:13:34 DEBUG } 23:13:34 DEBUG ] 23:13:34 DEBUG } 23:13:34 DEBUG } 23:13:34 INFO [http_cmd]: http status code OK 23:13:34 DEBUG --- status code --- 23:13:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:34 DEBUG --- http response --- 23:13:34 INFO [http_cmd]: http status code OK 23:13:34 DEBUG --- status code --- 23:13:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:34 DEBUG --- http response --- 23:13:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:34 DEBUG ----- output ----- 23:13:34 DEBUG { 23:13:34 DEBUG "status": "success", 23:13:34 DEBUG "data": { 23:13:34 DEBUG "resultType": "vector", 23:13:34 DEBUG "result": [ 23:13:34 DEBUG { 23:13:34 DEBUG "metric": { 23:13:34 DEBUG "instance": "10.142.0.107:9100" 23:13:34 DEBUG }, 23:13:34 DEBUG "value": [ 23:13:34 DEBUG ----- output ----- 23:13:34 DEBUG 23:13:34 DEBUG { 23:13:34 DEBUG 23:13:34 DEBUG "status": "success", 23:13:34 DEBUG 23:13:34 DEBUG "data": { 23:13:34 DEBUG [...] 23:13:34 DEBUG "resultType": "vector", 23:13:34 DEBUG 23:13:34 DEBUG "result": [ 23:13:34 DEBUG 23:13:34 DEBUG { 23:13:34 DEBUG "metric": { 23:13:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:13:34 DEBUG }, 23:13:34 DEBUG "value": [ 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG [...] 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG 23:13:34 DEBUG "instance": "10.142.0.43:9100" 23:13:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:13:34 DEBUG }, 23:13:34 DEBUG }, 23:13:34 DEBUG "value": [ 23:13:34 DEBUG "value": [ 23:13:34 DEBUG 1713392014, 23:13:34 DEBUG 1713392014, 23:13:34 DEBUG "2301435904" 23:13:34 DEBUG "60860459.13333333" 23:13:34 DEBUG ] 23:13:34 DEBUG ] 23:13:34 DEBUG } 23:13:34 DEBUG } 23:13:34 DEBUG ] 23:13:34 DEBUG ] 23:13:34 DEBUG } 23:13:34 DEBUG } 23:13:34 DEBUG } 23:13:34 DEBUG } 23:13:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392015" 23:13:35 INFO [http_cmd]: http status code OK 23:13:35 DEBUG --- status code --- 23:13:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:35 DEBUG --- http response --- 23:13:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:13:35 DEBUG ----- output ----- 23:13:35 DEBUG { 23:13:35 DEBUG "status": "success", 23:13:35 DEBUG "data": { 23:13:35 DEBUG "resultType": "vector", 23:13:35 DEBUG "result": [ 23:13:35 DEBUG { 23:13:35 DEBUG "metric": { 23:13:35 DEBUG "pod": "ds-cts-0" 23:13:35 DEBUG }, 23:13:35 DEBUG "value": [ 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG [...] 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG "pod": "ds-cts-2" 23:13:35 DEBUG }, 23:13:35 DEBUG "value": [ 23:13:35 DEBUG 1713392015, 23:13:35 DEBUG "29.933333333333334" 23:13:35 DEBUG ] 23:13:35 DEBUG } 23:13:35 DEBUG ] 23:13:35 DEBUG } 23:13:35 DEBUG } 23:13:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392015" 23:13:35 INFO [http_cmd]: http status code OK 23:13:35 DEBUG --- status code --- 23:13:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:35 DEBUG --- http response --- 23:13:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:35 DEBUG ----- output ----- 23:13:35 DEBUG { 23:13:35 DEBUG "status": "success", 23:13:35 DEBUG "data": { 23:13:35 DEBUG "resultType": "vector", 23:13:35 DEBUG "result": [ 23:13:35 DEBUG { 23:13:35 DEBUG "metric": { 23:13:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:13:35 DEBUG }, 23:13:35 DEBUG "value": [ 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG [...] 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:13:35 DEBUG }, 23:13:35 DEBUG "value": [ 23:13:35 DEBUG 1713392015, 23:13:35 DEBUG "1823.1333333333332" 23:13:35 DEBUG ] 23:13:35 DEBUG } 23:13:35 DEBUG ] 23:13:35 DEBUG } 23:13:35 DEBUG } 23:13:35 INFO 23:13:35 INFO [loop_until]: kubectl --namespace=xlou top pods 23:13:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:13:35 INFO [loop_until]: OK (rc = 0) 23:13:35 DEBUG --- stdout --- 23:13:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11895m 8915Mi am-b59c765b7-bgsxp 11112m 8770Mi am-b59c765b7-ch5q4 11948m 8536Mi ds-cts-0 12434m 7942Mi ds-cts-1 11698m 7782Mi ds-cts-2 13265m 8001Mi ds-idrepo-0 5442m 3221Mi ds-idrepo-1 5289m 2879Mi ds-idrepo-2 5203m 2933Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6431m 2103Mi 23:13:35 DEBUG --- stderr --- 23:13:35 DEBUG 23:13:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392015" 23:13:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392015" 23:13:35 INFO [http_cmd]: http status code OK 23:13:35 DEBUG --- status code --- 23:13:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:35 DEBUG --- http response --- 23:13:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:13:35 DEBUG ----- output ----- 23:13:35 DEBUG { 23:13:35 DEBUG "status": "success", 23:13:35 DEBUG "data": { 23:13:35 DEBUG "resultType": "vector", 23:13:35 DEBUG "result": [ 23:13:35 DEBUG { 23:13:35 DEBUG "metric": { 23:13:35 DEBUG "pod": "ds-idrepo-0" 23:13:35 DEBUG }, 23:13:35 DEBUG "value": [ 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG [...] 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG "pod": "am-b59c765b7-7xg4f" 23:13:35 DEBUG }, 23:13:35 DEBUG "value": [ 23:13:35 DEBUG 1713392015, 23:13:35 DEBUG "0" 23:13:35 DEBUG ] 23:13:35 DEBUG } 23:13:35 DEBUG ] 23:13:35 DEBUG } 23:13:35 DEBUG } 23:13:35 INFO [http_cmd]: http status code OK 23:13:35 DEBUG --- status code --- 23:13:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:35 DEBUG --- http response --- 23:13:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:35 DEBUG ----- output ----- 23:13:35 DEBUG { 23:13:35 DEBUG "status": "success", 23:13:35 DEBUG "data": { 23:13:35 DEBUG "resultType": "vector", 23:13:35 DEBUG "result": [ 23:13:35 DEBUG { 23:13:35 DEBUG "metric": { 23:13:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:13:35 DEBUG }, 23:13:35 DEBUG "value": [ 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG [...] 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG 23:13:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:13:35 DEBUG }, 23:13:35 DEBUG "value": [ 23:13:35 DEBUG 1713392015, 23:13:35 DEBUG "1814.9666666666667" 23:13:35 DEBUG ] 23:13:35 DEBUG } 23:13:35 DEBUG ] 23:13:35 DEBUG } 23:13:35 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392016" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "instance": "10.142.0.107:9100" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "instance": "10.142.0.43:9100" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG "240650834944" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392016" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392016" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "ds-idrepo-0" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG "0" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG "1814.6666666666667" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713392016" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713392016" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713392016" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392016" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392016" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392016" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392016" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392016" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392016" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392016" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392016" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392016" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392016" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG { 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG ----- output ----- 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG { 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG "data": { 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG "data": { 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG "result": [ 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG { 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "result": [ 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:13:36 DEBUG { 23:13:36 DEBUG { 23:13:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:13:36 DEBUG "pod": "ds-idrepo-0" 23:13:36 DEBUG }, 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG }, 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:13:36 DEBUG "data": { 23:13:36 DEBUG "value": [ 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:13:36 DEBUG "result": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "pod": "ds-idrepo-0" 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "data": { 23:13:36 DEBUG }, 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:13:36 DEBUG "pod": "ds-cts-2" 23:13:36 DEBUG }, 23:13:36 DEBUG "result": [ 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:13:36 DEBUG }, 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG { 23:13:36 DEBUG 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG "value": [ 23:13:36 INFO 23:13:36 DEBUG 1713392016, 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG "metric": { 23:13:36 DEBUG 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG 1713392016, 23:13:36 INFO [loop_until]: kubectl --namespace=xlou top node 23:13:36 DEBUG "6.615224181921828" 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG "pod": "ds-cts-0" 23:13:36 DEBUG [...] 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG "0.018" 23:13:36 DEBUG "0" 23:13:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:13:36 DEBUG ] 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- http response --- 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG }, 23:13:36 DEBUG 23:13:36 DEBUG "327554" 23:13:36 DEBUG ] 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG { 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG "pod": "ds-idrepo-2" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 DEBUG { 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:13:36 DEBUG 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG }, 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG "data": { 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG "value": [ 23:13:36 DEBUG } 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG { 23:13:36 DEBUG [...] 23:13:36 DEBUG "data": { 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG { 23:13:36 DEBUG "0" 23:13:36 DEBUG { 23:13:36 DEBUG "data": { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG { 23:13:36 DEBUG "data": { 23:13:36 DEBUG 23:13:36 DEBUG "result": [ 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG ] 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG 23:13:36 DEBUG { 23:13:36 DEBUG "data": { 23:13:36 DEBUG } 23:13:36 DEBUG "pod": "ds-idrepo-0" 23:13:36 DEBUG "data": { 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "result": [ 23:13:36 DEBUG "pod": "ds-cts-2" 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG ] 23:13:36 DEBUG }, 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "instance": "10.142.0.107:9100" 23:13:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:13:36 DEBUG { 23:13:36 DEBUG }, 23:13:36 DEBUG "pod": "ds-idrepo-0" 23:13:36 DEBUG "result": [ 23:13:36 DEBUG } 23:13:36 DEBUG "value": [ 23:13:36 DEBUG "result": [ 23:13:36 DEBUG }, 23:13:36 DEBUG }, 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "value": [ 23:13:36 DEBUG }, 23:13:36 DEBUG { 23:13:36 DEBUG } 23:13:36 DEBUG 23:13:36 DEBUG { 23:13:36 DEBUG "value": [ 23:13:36 DEBUG "value": [ 23:13:36 DEBUG "instance": "10.142.0.107:9100" 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG "metric": { 23:13:36 DEBUG 23:13:36 DEBUG "metric": { 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG }, 23:13:36 DEBUG "0" 23:13:36 DEBUG 23:13:36 DEBUG "pod": "ds-idrepo-0" 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG ] 23:13:36 DEBUG 23:13:36 DEBUG }, 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "value": [ 23:13:36 DEBUG "pod": "ds-cts-0" 23:13:36 DEBUG 23:13:36 DEBUG } 23:13:36 DEBUG 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG }, 23:13:36 DEBUG [...] 23:13:36 DEBUG ] 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG } 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG }, 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG } 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "ds-idrepo-2" 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG }, 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG "instance": "10.142.0.43:9100" 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG "0" 23:13:36 DEBUG "value": [ 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG }, 23:13:36 DEBUG 23:13:36 DEBUG }, 23:13:36 DEBUG ] 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG "0.0803333333333333" 23:13:36 DEBUG "value": [ 23:13:36 DEBUG "0" 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG ] 23:13:36 DEBUG 23:13:36 DEBUG "instance": "10.142.0.43:9100" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG "pod": "ds-idrepo-2" 23:13:36 DEBUG "0" 23:13:36 DEBUG } 23:13:36 DEBUG "pod": "ds-cts-2" 23:13:36 DEBUG }, 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG }, 23:13:36 DEBUG ] 23:13:36 DEBUG ] 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 DEBUG "value": [ 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG } 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG 1713392016, 23:13:36 DEBUG "60859846.3" 23:13:36 DEBUG "0" 23:13:36 DEBUG } 23:13:36 DEBUG "0.00009508444444444445" 23:13:36 DEBUG ] 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG ] 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [loop_until]: OK (rc = 0) 23:13:36 DEBUG --- stdout --- 23:13:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5448Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5219m 17% 6152Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11977m 40% 11116Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 2992Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12487m 41% 10194Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 4965Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11943m 40% 11512Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27984m 93% 2926Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11854m 39% 10617Mi 9% gke-xlou-cdm-ds-0f601751-j488 5656m 18% 5509Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11570m 38% 11352Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5835m 19% 6001Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13123m 43% 9717Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6790m 42% 5864Mi 9% 23:13:36 DEBUG --- stderr --- 23:13:36 DEBUG 23:14:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392074" 23:14:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713392074" 23:14:34 ERROR [http_cmd]: ERROR 23:14:34 INFO [http_cmd]: http status code OK 23:14:34 ERROR --- status code --- 23:14:34 DEBUG --- status code --- 23:14:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:14:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:34 ERROR --- http response --- 23:14:34 DEBUG --- http response --- 23:14:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:14:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:14:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:14:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392074" 23:14:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392074" 23:14:34 INFO [http_cmd]: http status code OK 23:14:34 DEBUG --- status code --- 23:14:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:34 DEBUG --- http response --- 23:14:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:34 DEBUG ----- output ----- 23:14:34 DEBUG { 23:14:34 DEBUG "status": "success", 23:14:34 DEBUG "data": { 23:14:34 DEBUG "resultType": "vector", 23:14:34 DEBUG "result": [ 23:14:34 DEBUG { 23:14:34 DEBUG "metric": { 23:14:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:14:34 DEBUG }, 23:14:34 DEBUG "value": [ 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG [...] 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:14:34 DEBUG }, 23:14:34 DEBUG "value": [ 23:14:34 DEBUG 1713392074, 23:14:34 DEBUG "1823.2" 23:14:34 DEBUG ] 23:14:34 DEBUG } 23:14:34 DEBUG ] 23:14:34 DEBUG } 23:14:34 DEBUG } 23:14:34 INFO [http_cmd]: http status code OK 23:14:34 DEBUG --- status code --- 23:14:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:34 DEBUG --- http response --- 23:14:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:14:34 DEBUG ----- output ----- 23:14:34 DEBUG { 23:14:34 DEBUG "status": "success", 23:14:34 DEBUG "data": { 23:14:34 DEBUG "resultType": "vector", 23:14:34 DEBUG "result": [ 23:14:34 DEBUG { 23:14:34 DEBUG "metric": { 23:14:34 DEBUG "pod": "ds-idrepo-0" 23:14:34 DEBUG }, 23:14:34 DEBUG "value": [ 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG [...] 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:14:34 DEBUG }, 23:14:34 DEBUG "value": [ 23:14:34 DEBUG 1713392074, 23:14:34 DEBUG "0.00002098212675932405" 23:14:34 DEBUG ] 23:14:34 DEBUG } 23:14:34 DEBUG ] 23:14:34 DEBUG } 23:14:34 DEBUG } 23:14:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392074" 23:14:34 INFO [http_cmd]: http status code OK 23:14:34 DEBUG --- status code --- 23:14:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:34 DEBUG --- http response --- 23:14:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:34 DEBUG ----- output ----- 23:14:34 DEBUG { 23:14:34 DEBUG "status": "success", 23:14:34 DEBUG "data": { 23:14:34 DEBUG "resultType": "vector", 23:14:34 DEBUG "result": [ 23:14:34 DEBUG { 23:14:34 DEBUG "metric": { 23:14:34 DEBUG "instance": "10.142.0.107:9100" 23:14:34 DEBUG }, 23:14:34 DEBUG "value": [ 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG [...] 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG "instance": "10.142.0.43:9100" 23:14:34 DEBUG }, 23:14:34 DEBUG "value": [ 23:14:34 DEBUG 1713392074, 23:14:34 DEBUG "0.14989999999999762" 23:14:34 DEBUG ] 23:14:34 DEBUG } 23:14:34 DEBUG ] 23:14:34 DEBUG } 23:14:34 DEBUG } 23:14:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713392074" 23:14:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392074" 23:14:34 INFO [http_cmd]: http status code OK 23:14:34 DEBUG --- status code --- 23:14:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:34 DEBUG --- http response --- 23:14:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:34 DEBUG ----- output ----- 23:14:34 INFO [http_cmd]: http status code OK 23:14:34 DEBUG { 23:14:34 DEBUG --- status code --- 23:14:34 DEBUG "status": "success", 23:14:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:34 DEBUG "data": { 23:14:34 DEBUG --- http response --- 23:14:34 DEBUG "resultType": "vector", 23:14:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:34 DEBUG "result": [ 23:14:34 DEBUG ----- output ----- 23:14:34 DEBUG { 23:14:34 DEBUG { 23:14:34 DEBUG "metric": { 23:14:34 DEBUG "status": "success", 23:14:34 DEBUG "instance": "10.142.0.107:9100" 23:14:34 DEBUG "data": { 23:14:34 DEBUG }, 23:14:34 DEBUG "resultType": "vector", 23:14:34 DEBUG "value": [ 23:14:34 DEBUG "result": [ 23:14:34 DEBUG 23:14:34 DEBUG { 23:14:34 DEBUG 23:14:34 DEBUG "metric": { 23:14:34 DEBUG 23:14:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:14:34 DEBUG [...] 23:14:34 DEBUG }, 23:14:34 DEBUG 23:14:34 DEBUG "value": [ 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG 23:14:34 DEBUG "instance": "10.142.0.43:9100" 23:14:34 DEBUG 23:14:34 DEBUG }, 23:14:34 DEBUG [...] 23:14:34 DEBUG "value": [ 23:14:34 DEBUG 23:14:34 DEBUG 1713392074, 23:14:34 DEBUG 23:14:34 DEBUG "60791094.4" 23:14:34 DEBUG 23:14:34 DEBUG ] 23:14:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:14:34 DEBUG } 23:14:34 DEBUG }, 23:14:34 DEBUG ] 23:14:34 DEBUG "value": [ 23:14:34 DEBUG } 23:14:34 DEBUG 1713392074, 23:14:34 DEBUG } 23:14:34 DEBUG "2303344640" 23:14:34 DEBUG ] 23:14:34 DEBUG } 23:14:34 DEBUG ] 23:14:34 DEBUG } 23:14:34 DEBUG } 23:14:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392075" 23:14:35 INFO [http_cmd]: http status code OK 23:14:35 DEBUG --- status code --- 23:14:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:35 DEBUG --- http response --- 23:14:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:14:35 DEBUG ----- output ----- 23:14:35 DEBUG { 23:14:35 DEBUG "status": "success", 23:14:35 DEBUG "data": { 23:14:35 DEBUG "resultType": "vector", 23:14:35 DEBUG "result": [ 23:14:35 DEBUG { 23:14:35 DEBUG "metric": { 23:14:35 DEBUG "pod": "ds-cts-0" 23:14:35 DEBUG }, 23:14:35 DEBUG "value": [ 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG [...] 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG "pod": "ds-cts-2" 23:14:35 DEBUG }, 23:14:35 DEBUG "value": [ 23:14:35 DEBUG 1713392075, 23:14:35 DEBUG "29.966666666666665" 23:14:35 DEBUG ] 23:14:35 DEBUG } 23:14:35 DEBUG ] 23:14:35 DEBUG } 23:14:35 DEBUG } 23:14:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392075" 23:14:35 INFO [http_cmd]: http status code OK 23:14:35 DEBUG --- status code --- 23:14:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:35 DEBUG --- http response --- 23:14:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:35 DEBUG ----- output ----- 23:14:35 DEBUG { 23:14:35 DEBUG "status": "success", 23:14:35 DEBUG "data": { 23:14:35 DEBUG "resultType": "vector", 23:14:35 DEBUG "result": [ 23:14:35 DEBUG { 23:14:35 DEBUG "metric": { 23:14:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:14:35 DEBUG }, 23:14:35 DEBUG "value": [ 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG [...] 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:14:35 DEBUG }, 23:14:35 DEBUG "value": [ 23:14:35 DEBUG 1713392075, 23:14:35 DEBUG "1824.0333333333333" 23:14:35 DEBUG ] 23:14:35 DEBUG } 23:14:35 DEBUG ] 23:14:35 DEBUG } 23:14:35 DEBUG } 23:14:35 INFO 23:14:35 INFO [loop_until]: kubectl --namespace=xlou top pods 23:14:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:14:35 INFO [loop_until]: OK (rc = 0) 23:14:35 DEBUG --- stdout --- 23:14:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 11899m 8914Mi am-b59c765b7-bgsxp 10707m 8771Mi am-b59c765b7-ch5q4 11913m 8537Mi ds-cts-0 12734m 7921Mi ds-cts-1 11373m 7789Mi ds-cts-2 12962m 7998Mi ds-idrepo-0 5317m 3221Mi ds-idrepo-1 5221m 2880Mi ds-idrepo-2 5280m 2932Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6467m 2104Mi 23:14:35 DEBUG --- stderr --- 23:14:35 DEBUG 23:14:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392075" 23:14:35 INFO [http_cmd]: http status code OK 23:14:35 DEBUG --- status code --- 23:14:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:35 DEBUG --- http response --- 23:14:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392075" 23:14:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:35 DEBUG ----- output ----- 23:14:35 DEBUG { 23:14:35 DEBUG "status": "success", 23:14:35 DEBUG "data": { 23:14:35 DEBUG "resultType": "vector", 23:14:35 DEBUG "result": [ 23:14:35 DEBUG { 23:14:35 DEBUG "metric": { 23:14:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:14:35 DEBUG }, 23:14:35 DEBUG "value": [ 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG [...] 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:14:35 DEBUG }, 23:14:35 DEBUG "value": [ 23:14:35 DEBUG 1713392075, 23:14:35 DEBUG "1822.7333333333333" 23:14:35 DEBUG ] 23:14:35 DEBUG } 23:14:35 DEBUG ] 23:14:35 DEBUG } 23:14:35 DEBUG } 23:14:35 INFO [http_cmd]: http status code OK 23:14:35 DEBUG --- status code --- 23:14:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:35 DEBUG --- http response --- 23:14:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:14:35 DEBUG ----- output ----- 23:14:35 DEBUG { 23:14:35 DEBUG "status": "success", 23:14:35 DEBUG "data": { 23:14:35 DEBUG "resultType": "vector", 23:14:35 DEBUG "result": [ 23:14:35 DEBUG { 23:14:35 DEBUG "metric": { 23:14:35 DEBUG "pod": "ds-idrepo-0" 23:14:35 DEBUG }, 23:14:35 DEBUG "value": [ 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG [...] 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG 23:14:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:14:35 DEBUG }, 23:14:35 DEBUG "value": [ 23:14:35 DEBUG 1713392075, 23:14:35 DEBUG "0" 23:14:35 DEBUG ] 23:14:35 DEBUG } 23:14:35 DEBUG ] 23:14:35 DEBUG } 23:14:35 DEBUG } 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392076" 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG { 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG "data": { 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG "result": [ 23:14:36 DEBUG { 23:14:36 DEBUG "metric": { 23:14:36 DEBUG "instance": "10.142.0.107:9100" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG [...] 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG "instance": "10.142.0.43:9100" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG "243677756416" 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG } 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392076" 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392076" 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG { 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG "data": { 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG "result": [ 23:14:36 DEBUG { 23:14:36 DEBUG "metric": { 23:14:36 DEBUG "pod": "ds-idrepo-0" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG [...] 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG "0" 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG } 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG { 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG "data": { 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG "result": [ 23:14:36 DEBUG { 23:14:36 DEBUG "metric": { 23:14:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG [...] 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG "1823.3" 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG } 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713392076" 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392076" 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG { 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG "data": { 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG "result": [ 23:14:36 DEBUG { 23:14:36 DEBUG "metric": { 23:14:36 DEBUG "pod": "ds-idrepo-0" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG [...] 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG "pod": "ds-cts-2" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG "0" 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG } 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713392076" 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG { 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG "data": { 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG "result": [ 23:14:36 DEBUG { 23:14:36 DEBUG "metric": { 23:14:36 DEBUG "instance": "10.142.0.107:9100" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG [...] 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG "instance": "10.142.0.43:9100" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG "0.07333333333333387" 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG } 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392076" 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713392076" 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG { 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG "data": { 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG "result": [ 23:14:36 DEBUG { 23:14:36 DEBUG "metric": { 23:14:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG [...] 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG "6.610122060240052" 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG } 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG { 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG "data": { 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG "result": [ 23:14:36 DEBUG { 23:14:36 DEBUG "metric": { 23:14:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG [...] 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG "327738" 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG } 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392076" 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392076" 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG --- http response --- 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392076" 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG { 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG "data": { 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG "result": [ 23:14:36 DEBUG { 23:14:36 DEBUG "metric": { 23:14:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG [...] 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG "0.018" 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG } 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392076" 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG { 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG "data": { 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG "result": [ 23:14:36 DEBUG { 23:14:36 DEBUG "metric": { 23:14:36 DEBUG "pod": "ds-idrepo-0" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG [...] 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG "pod": "ds-idrepo-2" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG "0" 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG } 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG --- http response --- 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392076" 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG { 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG "data": { 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG "result": [ 23:14:36 DEBUG { 23:14:36 DEBUG "metric": { 23:14:36 DEBUG "pod": "ds-cts-0" 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:36 DEBUG 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392076" 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG [...] 23:14:36 DEBUG --- status code --- 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392076" 23:14:36 DEBUG { 23:14:36 DEBUG 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG 23:14:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392076" 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG "data": { 23:14:36 DEBUG 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG "pod": "ds-cts-2" 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG "result": [ 23:14:36 DEBUG }, 23:14:36 DEBUG { 23:14:36 DEBUG { 23:14:36 DEBUG "value": [ 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG "metric": { 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG "data": { 23:14:36 DEBUG "instance": "10.142.0.107:9100" 23:14:36 DEBUG "0.00007131333333333333" 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG }, 23:14:36 DEBUG ] 23:14:36 DEBUG "result": [ 23:14:36 DEBUG "value": [ 23:14:36 DEBUG } 23:14:36 DEBUG { 23:14:36 DEBUG 23:14:36 DEBUG ] 23:14:36 DEBUG "metric": { 23:14:36 DEBUG 23:14:36 DEBUG } 23:14:36 DEBUG "pod": "ds-cts-0" 23:14:36 DEBUG 23:14:36 DEBUG } 23:14:36 DEBUG }, 23:14:36 DEBUG [...] 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG "instance": "10.142.0.43:9100" 23:14:36 DEBUG [...] 23:14:36 DEBUG }, 23:14:36 DEBUG 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG "60791875.03333333" 23:14:36 DEBUG "pod": "ds-cts-2" 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG ] 23:14:36 DEBUG }, 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG } 23:14:36 DEBUG "value": [ 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG ] 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG } 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG "0" 23:14:36 DEBUG } 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 INFO [http_cmd]: http status code OK 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG { 23:14:36 DEBUG ] 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG --- status code --- 23:14:36 DEBUG { 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG } 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG "data": { 23:14:36 DEBUG ] 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG --- http response --- 23:14:36 DEBUG "data": { 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG } 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:36 DEBUG "result": [ 23:14:36 DEBUG } 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG "result": [ 23:14:36 DEBUG ----- output ----- 23:14:36 DEBUG { 23:14:36 DEBUG { 23:14:36 DEBUG { 23:14:36 DEBUG { 23:14:36 DEBUG "metric": { 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG "metric": { 23:14:36 DEBUG "status": "success", 23:14:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:14:36 DEBUG "data": { 23:14:36 DEBUG "pod": "ds-idrepo-0" 23:14:36 DEBUG "data": { 23:14:36 DEBUG }, 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG }, 23:14:36 DEBUG "resultType": "vector", 23:14:36 DEBUG "value": [ 23:14:36 DEBUG "result": [ 23:14:36 DEBUG "value": [ 23:14:36 DEBUG "result": [ 23:14:36 DEBUG 23:14:36 DEBUG { 23:14:36 DEBUG 23:14:36 DEBUG { 23:14:36 DEBUG 23:14:36 DEBUG "metric": { 23:14:36 DEBUG 23:14:36 DEBUG "metric": { 23:14:36 DEBUG 23:14:36 DEBUG "pod": "ds-idrepo-0" 23:14:36 DEBUG 23:14:36 DEBUG "pod": "ds-idrepo-0" 23:14:36 DEBUG [...] 23:14:36 DEBUG }, 23:14:36 DEBUG "value": [ 23:14:36 DEBUG }, 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG [...] 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:14:36 DEBUG [...] 23:14:36 DEBUG 23:14:36 DEBUG 23:14:36 DEBUG }, 23:14:36 DEBUG 23:14:36 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:14:36 DEBUG [...] 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG }, 23:14:36 DEBUG 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG 23:14:36 DEBUG "value": [ 23:14:36 DEBUG 23:14:36 DEBUG "0" 23:14:36 DEBUG "pod": "ds-idrepo-2" 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG 23:14:36 DEBUG ] 23:14:36 DEBUG }, 23:14:36 DEBUG "0" 23:14:36 DEBUG "pod": "ds-idrepo-2" 23:14:36 DEBUG } 23:14:36 DEBUG "value": [ 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG ] 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG }, 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG } 23:14:36 DEBUG "value": [ 23:14:36 DEBUG } 23:14:36 DEBUG "0" 23:14:36 DEBUG 1713392076, 23:14:36 DEBUG } 23:14:36 DEBUG ] 23:14:36 DEBUG "0" 23:14:36 DEBUG } 23:14:36 DEBUG ] 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG } 23:14:36 DEBUG ] 23:14:36 DEBUG } 23:14:36 DEBUG } 23:14:36 DEBUG } 23:14:36 INFO 23:14:36 INFO [loop_until]: kubectl --namespace=xlou top node 23:14:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:14:37 INFO [loop_until]: OK (rc = 0) 23:14:37 DEBUG --- stdout --- 23:14:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5448Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5231m 17% 6153Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11389m 38% 11121Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 2986Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 12524m 41% 10192Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 4967Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11019m 36% 11519Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27053m 90% 2906Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 11751m 39% 10612Mi 9% gke-xlou-cdm-ds-0f601751-j488 5633m 18% 5516Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 11481m 38% 11364Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 5477m 18% 5996Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 13057m 43% 9716Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 6559m 41% 5868Mi 10% 23:14:37 DEBUG --- stderr --- 23:14:37 DEBUG 23:14:45 WARNING File /tmp/lodemon_data-20240417_224222/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [17/Apr/2024 23:14:45] "GET /monitoring/average?start_time=20240417_214534&stop_time=20240417_221245 HTTP/1.1" 200 - 23:15:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713392134" 23:15:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392134" 23:15:34 ERROR [http_cmd]: ERROR 23:15:34 ERROR --- status code --- 23:15:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:15:34 ERROR --- http response --- 23:15:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:15:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:15:34 INFO [http_cmd]: http status code OK 23:15:34 DEBUG --- status code --- 23:15:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:34 DEBUG --- http response --- 23:15:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:15:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392134" 23:15:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392134" 23:15:34 INFO [http_cmd]: http status code OK 23:15:34 DEBUG --- status code --- 23:15:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:34 DEBUG --- http response --- 23:15:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:34 DEBUG ----- output ----- 23:15:34 DEBUG { 23:15:34 DEBUG "status": "success", 23:15:34 DEBUG "data": { 23:15:34 DEBUG "resultType": "vector", 23:15:34 DEBUG "result": [ 23:15:34 DEBUG { 23:15:34 DEBUG "metric": { 23:15:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:15:34 DEBUG }, 23:15:34 DEBUG "value": [ 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG [...] 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:15:34 DEBUG }, 23:15:34 DEBUG "value": [ 23:15:34 DEBUG 1713392134, 23:15:34 DEBUG "102.2" 23:15:34 DEBUG ] 23:15:34 DEBUG } 23:15:34 DEBUG ] 23:15:34 DEBUG } 23:15:34 DEBUG } 23:15:34 INFO [http_cmd]: http status code OK 23:15:34 DEBUG --- status code --- 23:15:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:34 DEBUG --- http response --- 23:15:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:15:34 DEBUG ----- output ----- 23:15:34 DEBUG { 23:15:34 DEBUG "status": "success", 23:15:34 DEBUG "data": { 23:15:34 DEBUG "resultType": "vector", 23:15:34 DEBUG "result": [ 23:15:34 DEBUG { 23:15:34 DEBUG "metric": { 23:15:34 DEBUG "pod": "ds-idrepo-1" 23:15:34 DEBUG }, 23:15:34 DEBUG "value": [ 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG [...] 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG "pod": "ds-cts-1" 23:15:34 DEBUG }, 23:15:34 DEBUG "value": [ 23:15:34 DEBUG 1713392134, 23:15:34 DEBUG "2.5196976304560783" 23:15:34 DEBUG ] 23:15:34 DEBUG } 23:15:34 DEBUG ] 23:15:34 DEBUG } 23:15:34 DEBUG } 23:15:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392134" 23:15:34 INFO [http_cmd]: http status code OK 23:15:34 DEBUG --- status code --- 23:15:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:34 DEBUG --- http response --- 23:15:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392134" 23:15:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:34 DEBUG ----- output ----- 23:15:34 DEBUG { 23:15:34 DEBUG "status": "success", 23:15:34 DEBUG "data": { 23:15:34 DEBUG "resultType": "vector", 23:15:34 DEBUG "result": [ 23:15:34 DEBUG { 23:15:34 DEBUG "metric": { 23:15:34 DEBUG "instance": "10.142.0.107:9100" 23:15:34 DEBUG }, 23:15:34 DEBUG "value": [ 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG [...] 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG "instance": "10.142.0.43:9100" 23:15:34 DEBUG }, 23:15:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713392134" 23:15:34 DEBUG "value": [ 23:15:34 DEBUG 1713392134, 23:15:34 DEBUG "0.01553333333333266" 23:15:34 DEBUG ] 23:15:34 DEBUG } 23:15:34 DEBUG ] 23:15:34 DEBUG } 23:15:34 DEBUG } 23:15:34 INFO [http_cmd]: http status code OK 23:15:34 INFO [http_cmd]: http status code OK 23:15:34 DEBUG --- status code --- 23:15:34 DEBUG --- status code --- 23:15:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:34 DEBUG --- http response --- 23:15:34 DEBUG --- http response --- 23:15:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:34 DEBUG ----- output ----- 23:15:34 DEBUG { 23:15:34 DEBUG "status": "success", 23:15:34 DEBUG "data": { 23:15:34 DEBUG "resultType": "vector", 23:15:34 DEBUG "result": [ 23:15:34 DEBUG { 23:15:34 DEBUG "metric": { 23:15:34 DEBUG "instance": "10.142.0.107:9100" 23:15:34 DEBUG }, 23:15:34 DEBUG "value": [ 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG [...] 23:15:34 DEBUG 23:15:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:34 DEBUG 23:15:34 DEBUG ----- output ----- 23:15:34 DEBUG 23:15:34 DEBUG { 23:15:34 DEBUG "instance": "10.142.0.43:9100" 23:15:34 DEBUG "status": "success", 23:15:34 DEBUG }, 23:15:34 DEBUG "data": { 23:15:34 DEBUG "value": [ 23:15:34 DEBUG "resultType": "vector", 23:15:34 DEBUG 1713392134, 23:15:34 DEBUG "result": [ 23:15:34 DEBUG "13636399.533333333" 23:15:34 DEBUG { 23:15:34 DEBUG ] 23:15:34 DEBUG "metric": { 23:15:34 DEBUG } 23:15:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:15:34 DEBUG ] 23:15:34 DEBUG }, 23:15:34 DEBUG } 23:15:34 DEBUG "value": [ 23:15:34 DEBUG } 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG [...] 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG 23:15:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:15:34 DEBUG }, 23:15:34 DEBUG "value": [ 23:15:34 DEBUG 1713392134, 23:15:34 DEBUG "2045173760" 23:15:34 DEBUG ] 23:15:34 DEBUG } 23:15:34 DEBUG ] 23:15:34 DEBUG } 23:15:34 DEBUG } 23:15:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392135" 23:15:35 INFO [http_cmd]: http status code OK 23:15:35 DEBUG --- status code --- 23:15:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:35 DEBUG --- http response --- 23:15:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:15:35 DEBUG ----- output ----- 23:15:35 DEBUG { 23:15:35 DEBUG "status": "success", 23:15:35 DEBUG "data": { 23:15:35 DEBUG "resultType": "vector", 23:15:35 DEBUG "result": [ 23:15:35 DEBUG { 23:15:35 DEBUG "metric": { 23:15:35 DEBUG "pod": "ds-cts-0" 23:15:35 DEBUG }, 23:15:35 DEBUG "value": [ 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG [...] 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG "pod": "ds-cts-2" 23:15:35 DEBUG }, 23:15:35 DEBUG "value": [ 23:15:35 DEBUG 1713392135, 23:15:35 DEBUG "29.933333333333334" 23:15:35 DEBUG ] 23:15:35 DEBUG } 23:15:35 DEBUG ] 23:15:35 DEBUG } 23:15:35 DEBUG } 23:15:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392135" 23:15:35 INFO [http_cmd]: http status code OK 23:15:35 DEBUG --- status code --- 23:15:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:35 DEBUG --- http response --- 23:15:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:35 DEBUG ----- output ----- 23:15:35 DEBUG { 23:15:35 DEBUG "status": "success", 23:15:35 DEBUG "data": { 23:15:35 DEBUG "resultType": "vector", 23:15:35 DEBUG "result": [ 23:15:35 DEBUG { 23:15:35 DEBUG "metric": { 23:15:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:15:35 DEBUG }, 23:15:35 DEBUG "value": [ 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG [...] 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:15:35 DEBUG }, 23:15:35 DEBUG "value": [ 23:15:35 DEBUG 1713392135, 23:15:35 DEBUG "1829.8333333333333" 23:15:35 DEBUG ] 23:15:35 DEBUG } 23:15:35 DEBUG ] 23:15:35 DEBUG } 23:15:35 DEBUG } 23:15:35 INFO 23:15:35 INFO [loop_until]: kubectl --namespace=xlou top pods 23:15:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:15:35 INFO [loop_until]: OK (rc = 0) 23:15:35 DEBUG --- stdout --- 23:15:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 2780m 7378Mi am-b59c765b7-bgsxp 2812m 7746Mi am-b59c765b7-ch5q4 2413m 8535Mi ds-cts-0 6275m 7995Mi ds-cts-1 5729m 7760Mi ds-cts-2 7455m 8006Mi ds-idrepo-0 977m 3221Mi ds-idrepo-1 1359m 2879Mi ds-idrepo-2 1298m 2932Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 3m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 3108m 1923Mi 23:15:35 DEBUG --- stderr --- 23:15:35 DEBUG 23:15:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392135" 23:15:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392135" 23:15:35 INFO [http_cmd]: http status code OK 23:15:35 DEBUG --- status code --- 23:15:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:35 DEBUG --- http response --- 23:15:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:35 DEBUG ----- output ----- 23:15:35 DEBUG { 23:15:35 DEBUG "status": "success", 23:15:35 DEBUG "data": { 23:15:35 DEBUG "resultType": "vector", 23:15:35 DEBUG "result": [ 23:15:35 DEBUG { 23:15:35 DEBUG "metric": { 23:15:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:15:35 DEBUG }, 23:15:35 DEBUG "value": [ 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG [...] 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:15:35 DEBUG }, 23:15:35 DEBUG "value": [ 23:15:35 DEBUG 1713392135, 23:15:35 DEBUG "102.3" 23:15:35 DEBUG ] 23:15:35 DEBUG } 23:15:35 DEBUG ] 23:15:35 DEBUG } 23:15:35 DEBUG } 23:15:35 INFO [http_cmd]: http status code OK 23:15:35 DEBUG --- status code --- 23:15:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:35 DEBUG --- http response --- 23:15:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:15:35 DEBUG ----- output ----- 23:15:35 DEBUG { 23:15:35 DEBUG "status": "success", 23:15:35 DEBUG "data": { 23:15:35 DEBUG "resultType": "vector", 23:15:35 DEBUG "result": [ 23:15:35 DEBUG { 23:15:35 DEBUG "metric": { 23:15:35 DEBUG "pod": "ds-idrepo-0" 23:15:35 DEBUG }, 23:15:35 DEBUG "value": [ 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG [...] 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG 23:15:35 DEBUG "pod": "ds-cts-1" 23:15:35 DEBUG }, 23:15:35 DEBUG "value": [ 23:15:35 DEBUG 1713392135, 23:15:35 DEBUG "4946.374797241379" 23:15:35 DEBUG ] 23:15:35 DEBUG } 23:15:35 DEBUG ] 23:15:35 DEBUG } 23:15:35 DEBUG } 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392136" 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG { 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG "data": { 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG "result": [ 23:15:36 DEBUG { 23:15:36 DEBUG "metric": { 23:15:36 DEBUG "instance": "10.142.0.107:9100" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG "instance": "10.142.0.43:9100" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "245073134592" 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392136" 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG { 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG "data": { 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG "result": [ 23:15:36 DEBUG { 23:15:36 DEBUG "metric": { 23:15:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "102" 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392136" 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG { 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG "data": { 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG "result": [ 23:15:36 DEBUG { 23:15:36 DEBUG "metric": { 23:15:36 DEBUG "pod": "ds-idrepo-0" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG "pod": "ds-cts-1" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "0" 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713392136" 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713392136" 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG { 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG "data": { 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG "result": [ 23:15:36 DEBUG { 23:15:36 DEBUG "metric": { 23:15:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "0.9867812641824731" 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 DEBUG } 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG } 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG { 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG "data": { 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG "result": [ 23:15:36 DEBUG { 23:15:36 DEBUG "metric": { 23:15:36 DEBUG "pod": "ds-idrepo-0" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG "pod": "ds-cts-2" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "0" 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392136" 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392136" 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713392136" 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG { 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG "data": { 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG "result": [ 23:15:36 DEBUG { 23:15:36 DEBUG "metric": { 23:15:36 DEBUG "instance": "10.142.0.107:9100" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG "instance": "10.142.0.43:9100" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "0.00866666666666654" 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392136" 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG { 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG "data": { 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG "result": [ 23:15:36 DEBUG { 23:15:36 DEBUG "metric": { 23:15:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "263237" 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:36 DEBUG { 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG { 23:15:36 DEBUG "data": { 23:15:36 DEBUG "status": "success", 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392136" 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG "data": { 23:15:36 DEBUG "result": [ 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG { 23:15:36 DEBUG "result": [ 23:15:36 DEBUG "metric": { 23:15:36 DEBUG { 23:15:36 DEBUG "pod": "ds-idrepo-0" 23:15:36 DEBUG "metric": { 23:15:36 DEBUG }, 23:15:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:15:36 DEBUG "value": [ 23:15:36 DEBUG }, 23:15:36 DEBUG 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392136" 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG "pod": "ds-idrepo-2" 23:15:36 DEBUG 23:15:36 DEBUG }, 23:15:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:15:36 DEBUG "value": [ 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392136" 23:15:36 DEBUG }, 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG "0" 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG ] 23:15:36 DEBUG "0.018" 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG { 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG "data": { 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG "result": [ 23:15:36 DEBUG { 23:15:36 DEBUG "metric": { 23:15:36 DEBUG "pod": "ds-cts-0" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG "pod": "ds-cts-2" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "0.00006666666666666667" 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG { 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG "data": { 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG "result": [ 23:15:36 DEBUG { 23:15:36 DEBUG "metric": { 23:15:36 DEBUG "instance": "10.142.0.107:9100" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG "instance": "10.142.0.43:9100" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "13633656.799999999" 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392136" 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392136" 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392136" 23:15:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392136" 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG { 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG "data": { 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG "result": [ 23:15:36 DEBUG { 23:15:36 DEBUG "metric": { 23:15:36 DEBUG "pod": "ds-cts-0" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG "pod": "ds-cts-2" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "0" 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG { 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG "data": { 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG "result": [ 23:15:36 DEBUG { 23:15:36 DEBUG "metric": { 23:15:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 23:15:36 DEBUG 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:15:36 DEBUG { 23:15:36 DEBUG }, 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG "value": [ 23:15:36 DEBUG "data": { 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "result": [ 23:15:36 DEBUG "0" 23:15:36 DEBUG { 23:15:36 DEBUG ] 23:15:36 DEBUG "metric": { 23:15:36 DEBUG } 23:15:36 DEBUG "pod": "ds-idrepo-0" 23:15:36 DEBUG ] 23:15:36 DEBUG }, 23:15:36 DEBUG } 23:15:36 DEBUG "value": [ 23:15:36 DEBUG } 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG "pod": "ds-cts-1" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "2374003.9751797705" 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG --- http response --- 23:15:36 INFO [http_cmd]: http status code OK 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:36 DEBUG --- status code --- 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:36 DEBUG { 23:15:36 DEBUG --- http response --- 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG "data": { 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG "result": [ 23:15:36 DEBUG { 23:15:36 DEBUG "metric": { 23:15:36 DEBUG "pod": "ds-idrepo-0" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:36 DEBUG 23:15:36 DEBUG ----- output ----- 23:15:36 DEBUG 23:15:36 DEBUG { 23:15:36 DEBUG "pod": "ds-idrepo-2" 23:15:36 DEBUG "status": "success", 23:15:36 DEBUG }, 23:15:36 DEBUG "data": { 23:15:36 DEBUG "value": [ 23:15:36 DEBUG "resultType": "vector", 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "result": [ 23:15:36 DEBUG "0" 23:15:36 DEBUG { 23:15:36 DEBUG ] 23:15:36 DEBUG "metric": { 23:15:36 DEBUG } 23:15:36 DEBUG "pod": "ds-idrepo-0" 23:15:36 DEBUG ] 23:15:36 DEBUG }, 23:15:36 DEBUG } 23:15:36 DEBUG "value": [ 23:15:36 DEBUG } 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG [...] 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG 23:15:36 DEBUG "pod": "ds-idrepo-2" 23:15:36 DEBUG }, 23:15:36 DEBUG "value": [ 23:15:36 DEBUG 1713392136, 23:15:36 DEBUG "0" 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG ] 23:15:36 DEBUG } 23:15:36 DEBUG } 23:15:37 INFO 23:15:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:15:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:15:37 INFO [loop_until]: OK (rc = 0) 23:15:37 DEBUG --- stdout --- 23:15:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 5448Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1333m 4% 6157Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 4306m 14% 11123Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 2995Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 7051m 23% 10277Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 4950Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 4719m 15% 9993Mi 8% gke-xlou-cdm-ds-0f601751-cf7b 2574m 8% 2889Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 3639m 12% 9600Mi 8% gke-xlou-cdm-ds-0f601751-j488 1081m 3% 5512Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 5826m 19% 11366Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 1805m 6% 6002Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 6813m 22% 9723Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 3335m 20% 5687Mi 9% 23:15:37 DEBUG --- stderr --- 23:15:37 DEBUG 23:16:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713392194" 23:16:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392194" 23:16:34 ERROR [http_cmd]: ERROR 23:16:34 ERROR --- status code --- 23:16:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:16:34 ERROR --- http response --- 23:16:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:16:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:16:34 INFO [http_cmd]: http status code OK 23:16:34 DEBUG --- status code --- 23:16:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:34 DEBUG --- http response --- 23:16:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:16:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392194" 23:16:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392194" 23:16:34 INFO [http_cmd]: http status code OK 23:16:34 DEBUG --- status code --- 23:16:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:34 DEBUG --- http response --- 23:16:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:34 DEBUG ----- output ----- 23:16:34 DEBUG { 23:16:34 DEBUG "status": "success", 23:16:34 DEBUG "data": { 23:16:34 DEBUG "resultType": "vector", 23:16:34 DEBUG "result": [ 23:16:34 DEBUG { 23:16:34 DEBUG "metric": { 23:16:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:16:34 DEBUG }, 23:16:34 DEBUG "value": [ 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG [...] 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:16:34 DEBUG }, 23:16:34 DEBUG "value": [ 23:16:34 DEBUG 1713392194, 23:16:34 DEBUG "1355.5333333333333" 23:16:34 DEBUG ] 23:16:34 DEBUG } 23:16:34 DEBUG ] 23:16:34 DEBUG } 23:16:34 DEBUG } 23:16:34 INFO [http_cmd]: http status code OK 23:16:34 DEBUG --- status code --- 23:16:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:34 DEBUG --- http response --- 23:16:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:16:34 DEBUG ----- output ----- 23:16:34 DEBUG { 23:16:34 DEBUG "status": "success", 23:16:34 DEBUG "data": { 23:16:34 DEBUG "resultType": "vector", 23:16:34 DEBUG "result": [ 23:16:34 DEBUG { 23:16:34 DEBUG "metric": { 23:16:34 DEBUG "pod": "ds-idrepo-1" 23:16:34 DEBUG }, 23:16:34 DEBUG "value": [ 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG [...] 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:16:34 DEBUG }, 23:16:34 DEBUG "value": [ 23:16:34 DEBUG 1713392194, 23:16:34 DEBUG "0.000019759674008307982" 23:16:34 DEBUG ] 23:16:34 DEBUG } 23:16:34 DEBUG ] 23:16:34 DEBUG } 23:16:34 DEBUG } 23:16:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392194" 23:16:34 INFO [http_cmd]: http status code OK 23:16:34 DEBUG --- status code --- 23:16:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:34 DEBUG --- http response --- 23:16:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:34 DEBUG ----- output ----- 23:16:34 DEBUG { 23:16:34 DEBUG "status": "success", 23:16:34 DEBUG "data": { 23:16:34 DEBUG "resultType": "vector", 23:16:34 DEBUG "result": [ 23:16:34 DEBUG { 23:16:34 DEBUG "metric": { 23:16:34 DEBUG "instance": "10.142.0.107:9100" 23:16:34 DEBUG }, 23:16:34 DEBUG "value": [ 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG [...] 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG "instance": "10.142.0.43:9100" 23:16:34 DEBUG }, 23:16:34 DEBUG "value": [ 23:16:34 DEBUG 1713392194, 23:16:34 DEBUG "0.05319999999999917" 23:16:34 DEBUG ] 23:16:34 DEBUG } 23:16:34 DEBUG ] 23:16:34 DEBUG } 23:16:34 DEBUG } 23:16:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392194" 23:16:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713392194" 23:16:34 INFO [http_cmd]: http status code OK 23:16:34 DEBUG --- status code --- 23:16:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:34 DEBUG --- http response --- 23:16:34 INFO [http_cmd]: http status code OK 23:16:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:34 DEBUG --- status code --- 23:16:34 DEBUG ----- output ----- 23:16:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:34 DEBUG { 23:16:34 DEBUG --- http response --- 23:16:34 DEBUG "status": "success", 23:16:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:34 DEBUG "data": { 23:16:34 DEBUG ----- output ----- 23:16:34 DEBUG "resultType": "vector", 23:16:34 DEBUG { 23:16:34 DEBUG "result": [ 23:16:34 DEBUG "status": "success", 23:16:34 DEBUG { 23:16:34 DEBUG "data": { 23:16:34 DEBUG "metric": { 23:16:34 DEBUG "resultType": "vector", 23:16:34 DEBUG "instance": "10.142.0.107:9100" 23:16:34 DEBUG "result": [ 23:16:34 DEBUG }, 23:16:34 DEBUG { 23:16:34 DEBUG "value": [ 23:16:34 DEBUG "metric": { 23:16:34 DEBUG 23:16:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:16:34 DEBUG 23:16:34 DEBUG }, 23:16:34 DEBUG 23:16:34 DEBUG "value": [ 23:16:34 DEBUG [...] 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG 23:16:34 DEBUG [...] 23:16:34 DEBUG "instance": "10.142.0.43:9100" 23:16:34 DEBUG 23:16:34 DEBUG }, 23:16:34 DEBUG 23:16:34 DEBUG "value": [ 23:16:34 DEBUG 23:16:34 DEBUG 1713392194, 23:16:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:16:34 DEBUG "51923722.6" 23:16:34 DEBUG }, 23:16:34 DEBUG ] 23:16:34 DEBUG "value": [ 23:16:34 DEBUG } 23:16:34 DEBUG 1713392194, 23:16:34 DEBUG ] 23:16:34 DEBUG "3356463104" 23:16:34 DEBUG } 23:16:34 DEBUG ] 23:16:34 DEBUG } 23:16:34 DEBUG } 23:16:34 DEBUG ] 23:16:34 DEBUG } 23:16:34 DEBUG } 23:16:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392195" 23:16:35 INFO [http_cmd]: http status code OK 23:16:35 DEBUG --- status code --- 23:16:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:35 DEBUG --- http response --- 23:16:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:16:35 DEBUG ----- output ----- 23:16:35 DEBUG { 23:16:35 DEBUG "status": "success", 23:16:35 DEBUG "data": { 23:16:35 DEBUG "resultType": "vector", 23:16:35 DEBUG "result": [ 23:16:35 DEBUG { 23:16:35 DEBUG "metric": { 23:16:35 DEBUG "pod": "ds-cts-0" 23:16:35 DEBUG }, 23:16:35 DEBUG "value": [ 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG [...] 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG "pod": "ds-cts-2" 23:16:35 DEBUG }, 23:16:35 DEBUG "value": [ 23:16:35 DEBUG 1713392195, 23:16:35 DEBUG "29.933333333333334" 23:16:35 DEBUG ] 23:16:35 DEBUG } 23:16:35 DEBUG ] 23:16:35 DEBUG } 23:16:35 DEBUG } 23:16:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392195" 23:16:35 INFO [http_cmd]: http status code OK 23:16:35 DEBUG --- status code --- 23:16:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:35 DEBUG --- http response --- 23:16:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:35 DEBUG ----- output ----- 23:16:35 DEBUG { 23:16:35 DEBUG "status": "success", 23:16:35 DEBUG "data": { 23:16:35 DEBUG "resultType": "vector", 23:16:35 DEBUG "result": [ 23:16:35 DEBUG { 23:16:35 DEBUG "metric": { 23:16:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:16:35 DEBUG }, 23:16:35 DEBUG "value": [ 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG [...] 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:16:35 DEBUG }, 23:16:35 DEBUG "value": [ 23:16:35 DEBUG 1713392195, 23:16:35 DEBUG "1813.9333333333334" 23:16:35 DEBUG ] 23:16:35 DEBUG } 23:16:35 DEBUG ] 23:16:35 DEBUG } 23:16:35 DEBUG } 23:16:35 INFO 23:16:35 INFO [loop_until]: kubectl --namespace=xlou top pods 23:16:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:16:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392195" 23:16:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392195" 23:16:35 INFO [http_cmd]: http status code OK 23:16:35 DEBUG --- status code --- 23:16:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:35 DEBUG --- http response --- 23:16:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:35 DEBUG ----- output ----- 23:16:35 DEBUG { 23:16:35 DEBUG "status": "success", 23:16:35 DEBUG "data": { 23:16:35 DEBUG "resultType": "vector", 23:16:35 DEBUG "result": [ 23:16:35 DEBUG { 23:16:35 DEBUG "metric": { 23:16:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:16:35 DEBUG }, 23:16:35 DEBUG "value": [ 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG [...] 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:16:35 DEBUG }, 23:16:35 DEBUG "value": [ 23:16:35 DEBUG 1713392195, 23:16:35 DEBUG "1355.4333333333334" 23:16:35 DEBUG ] 23:16:35 DEBUG } 23:16:35 DEBUG ] 23:16:35 DEBUG } 23:16:35 DEBUG } 23:16:35 INFO [http_cmd]: http status code OK 23:16:35 DEBUG --- status code --- 23:16:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:35 DEBUG --- http response --- 23:16:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:16:35 DEBUG ----- output ----- 23:16:35 DEBUG { 23:16:35 DEBUG "status": "success", 23:16:35 DEBUG "data": { 23:16:35 DEBUG "resultType": "vector", 23:16:35 DEBUG "result": [ 23:16:35 DEBUG { 23:16:35 DEBUG "metric": { 23:16:35 DEBUG "pod": "ds-idrepo-1" 23:16:35 DEBUG }, 23:16:35 DEBUG "value": [ 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG [...] 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG 23:16:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:16:35 DEBUG }, 23:16:35 DEBUG "value": [ 23:16:35 DEBUG 1713392195, 23:16:35 DEBUG "108.11772028096382" 23:16:35 DEBUG ] 23:16:35 DEBUG } 23:16:35 DEBUG ] 23:16:35 DEBUG } 23:16:35 DEBUG } 23:16:36 INFO [loop_until]: OK (rc = 0) 23:16:36 DEBUG --- stdout --- 23:16:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 8462m 7391Mi am-b59c765b7-bgsxp 7988m 7756Mi am-b59c765b7-ch5q4 8310m 8544Mi ds-cts-0 9725m 8016Mi ds-cts-1 9452m 7792Mi ds-cts-2 10715m 8021Mi ds-idrepo-0 4042m 3222Mi ds-idrepo-1 3781m 2879Mi ds-idrepo-2 3687m 2935Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 3m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8086m 4567Mi 23:16:36 DEBUG --- stderr --- 23:16:36 DEBUG 23:16:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392196" 23:16:36 INFO [http_cmd]: http status code OK 23:16:36 DEBUG --- status code --- 23:16:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:36 DEBUG --- http response --- 23:16:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:36 DEBUG ----- output ----- 23:16:36 DEBUG { 23:16:36 DEBUG "status": "success", 23:16:36 DEBUG "data": { 23:16:36 DEBUG "resultType": "vector", 23:16:36 DEBUG "result": [ 23:16:36 DEBUG { 23:16:36 DEBUG "metric": { 23:16:36 DEBUG "instance": "10.142.0.107:9100" 23:16:36 DEBUG }, 23:16:36 DEBUG "value": [ 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG [...] 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG "instance": "10.142.0.43:9100" 23:16:36 DEBUG }, 23:16:36 DEBUG "value": [ 23:16:36 DEBUG 1713392196, 23:16:36 DEBUG "247128978432" 23:16:36 DEBUG ] 23:16:36 DEBUG } 23:16:36 DEBUG ] 23:16:36 DEBUG } 23:16:36 DEBUG } 23:16:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392196" 23:16:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392196" 23:16:36 INFO [http_cmd]: http status code OK 23:16:36 DEBUG --- status code --- 23:16:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:36 DEBUG --- http response --- 23:16:36 INFO [http_cmd]: http status code OK 23:16:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:16:36 DEBUG --- status code --- 23:16:36 DEBUG ----- output ----- 23:16:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:36 DEBUG { 23:16:36 DEBUG --- http response --- 23:16:36 DEBUG "status": "success", 23:16:36 DEBUG "data": { 23:16:36 DEBUG "resultType": "vector", 23:16:36 DEBUG "result": [ 23:16:36 DEBUG { 23:16:36 DEBUG "metric": { 23:16:36 DEBUG "pod": "ds-idrepo-1" 23:16:36 DEBUG }, 23:16:36 DEBUG "value": [ 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG [...] 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:16:36 DEBUG }, 23:16:36 DEBUG "value": [ 23:16:36 DEBUG 1713392196, 23:16:36 DEBUG "0" 23:16:36 DEBUG ] 23:16:36 DEBUG } 23:16:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:36 DEBUG ] 23:16:36 DEBUG ----- output ----- 23:16:36 DEBUG } 23:16:36 DEBUG { 23:16:36 DEBUG } 23:16:36 DEBUG "status": "success", 23:16:36 DEBUG "data": { 23:16:36 DEBUG "resultType": "vector", 23:16:36 DEBUG "result": [ 23:16:36 DEBUG { 23:16:36 DEBUG "metric": { 23:16:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:16:36 DEBUG }, 23:16:36 DEBUG "value": [ 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG [...] 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:16:36 DEBUG }, 23:16:36 DEBUG "value": [ 23:16:36 DEBUG 1713392196, 23:16:36 DEBUG "1356.1333333333332" 23:16:36 DEBUG ] 23:16:36 DEBUG } 23:16:36 DEBUG ] 23:16:36 DEBUG } 23:16:36 DEBUG } 23:16:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713392196" 23:16:36 INFO [http_cmd]: http status code OK 23:16:36 DEBUG --- status code --- 23:16:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:36 DEBUG --- http response --- 23:16:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:36 DEBUG ----- output ----- 23:16:36 DEBUG { 23:16:36 DEBUG "status": "success", 23:16:36 DEBUG "data": { 23:16:36 DEBUG "resultType": "vector", 23:16:36 DEBUG "result": [ 23:16:36 DEBUG { 23:16:36 DEBUG "metric": { 23:16:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:16:36 DEBUG }, 23:16:36 DEBUG "value": [ 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG [...] 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:16:36 DEBUG }, 23:16:36 DEBUG "value": [ 23:16:36 DEBUG 1713392196, 23:16:36 DEBUG "6.864245249496833" 23:16:36 DEBUG ] 23:16:36 DEBUG } 23:16:36 DEBUG ] 23:16:36 DEBUG } 23:16:36 DEBUG } 23:16:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713392196" 23:16:36 INFO [http_cmd]: http status code OK 23:16:36 DEBUG --- status code --- 23:16:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:36 DEBUG --- http response --- 23:16:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:16:36 DEBUG ----- output ----- 23:16:36 DEBUG { 23:16:36 DEBUG "status": "success", 23:16:36 DEBUG "data": { 23:16:36 DEBUG "resultType": "vector", 23:16:36 DEBUG "result": [ 23:16:36 DEBUG { 23:16:36 DEBUG "metric": { 23:16:36 DEBUG "pod": "ds-idrepo-0" 23:16:36 DEBUG }, 23:16:36 DEBUG "value": [ 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG [...] 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG 23:16:36 DEBUG "pod": "ds-cts-2" 23:16:36 DEBUG }, 23:16:36 DEBUG "value": [ 23:16:36 DEBUG 1713392196, 23:16:36 DEBUG "0" 23:16:36 DEBUG ] 23:16:36 DEBUG } 23:16:36 DEBUG ] 23:16:36 DEBUG } 23:16:36 DEBUG } 23:16:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392196" 23:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392197" 23:16:37 INFO [http_cmd]: http status code OK 23:16:37 DEBUG --- status code --- 23:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:37 DEBUG --- http response --- 23:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:37 DEBUG ----- output ----- 23:16:37 DEBUG { 23:16:37 DEBUG "status": "success", 23:16:37 DEBUG "data": { 23:16:37 DEBUG "resultType": "vector", 23:16:37 DEBUG "result": [ 23:16:37 DEBUG { 23:16:37 DEBUG "metric": { 23:16:37 DEBUG "instance": "10.142.0.107:9100" 23:16:37 DEBUG }, 23:16:37 DEBUG "value": [ 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG [...] 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG "instance": "10.142.0.43:9100" 23:16:37 DEBUG }, 23:16:37 DEBUG "value": [ 23:16:37 DEBUG 1713392196, 23:16:37 DEBUG "0.03566666666666659" 23:16:37 DEBUG ] 23:16:37 DEBUG } 23:16:37 DEBUG ] 23:16:37 DEBUG } 23:16:37 DEBUG } 23:16:37 INFO [http_cmd]: http status code OK 23:16:37 DEBUG --- status code --- 23:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:37 DEBUG --- http response --- 23:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:37 DEBUG ----- output ----- 23:16:37 DEBUG { 23:16:37 DEBUG "status": "success", 23:16:37 DEBUG "data": { 23:16:37 DEBUG "resultType": "vector", 23:16:37 DEBUG "result": [ 23:16:37 DEBUG { 23:16:37 DEBUG "metric": { 23:16:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:16:37 DEBUG }, 23:16:37 DEBUG "value": [ 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG [...] 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:16:37 DEBUG }, 23:16:37 DEBUG "value": [ 23:16:37 DEBUG 1713392197, 23:16:37 DEBUG "219401" 23:16:37 DEBUG ] 23:16:37 DEBUG } 23:16:37 DEBUG ] 23:16:37 DEBUG } 23:16:37 DEBUG } 23:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392197" 23:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392197" 23:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713392197" 23:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392197" 23:16:37 INFO [http_cmd]: http status code OK 23:16:37 DEBUG --- status code --- 23:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:37 DEBUG --- http response --- 23:16:37 INFO [http_cmd]: http status code OK 23:16:37 DEBUG --- status code --- 23:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:37 DEBUG --- http response --- 23:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:37 DEBUG ----- output ----- 23:16:37 DEBUG { 23:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:37 DEBUG "status": "success", 23:16:37 DEBUG ----- output ----- 23:16:37 DEBUG "data": { 23:16:37 DEBUG { 23:16:37 DEBUG "resultType": "vector", 23:16:37 DEBUG "status": "success", 23:16:37 DEBUG "result": [ 23:16:37 DEBUG "data": { 23:16:37 DEBUG { 23:16:37 DEBUG "resultType": "vector", 23:16:37 DEBUG "metric": { 23:16:37 DEBUG "result": [ 23:16:37 DEBUG "pod": "ds-idrepo-0" 23:16:37 DEBUG { 23:16:37 DEBUG }, 23:16:37 DEBUG "metric": { 23:16:37 DEBUG "value": [ 23:16:37 DEBUG "instance": "10.142.0.107:9100" 23:16:37 DEBUG 23:16:37 DEBUG }, 23:16:37 DEBUG 23:16:37 DEBUG "value": [ 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG [...] 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392197" 23:16:37 DEBUG [...] 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG "pod": "ds-idrepo-2" 23:16:37 DEBUG 23:16:37 DEBUG }, 23:16:37 DEBUG 23:16:37 DEBUG "value": [ 23:16:37 DEBUG "instance": "10.142.0.43:9100" 23:16:37 DEBUG 1713392197, 23:16:37 DEBUG }, 23:16:37 DEBUG "0" 23:16:37 DEBUG "value": [ 23:16:37 DEBUG ] 23:16:37 DEBUG 1713392197, 23:16:37 INFO [http_cmd]: http status code OK 23:16:37 DEBUG } 23:16:37 DEBUG "51922784.3" 23:16:37 DEBUG --- status code --- 23:16:37 DEBUG ] 23:16:37 DEBUG ] 23:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:37 DEBUG } 23:16:37 INFO [http_cmd]: http status code OK 23:16:37 DEBUG } 23:16:37 DEBUG --- http response --- 23:16:37 DEBUG } 23:16:37 DEBUG --- status code --- 23:16:37 DEBUG ] 23:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:37 DEBUG } 23:16:37 DEBUG ----- output ----- 23:16:37 DEBUG --- http response --- 23:16:37 DEBUG } 23:16:37 DEBUG { 23:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:16:37 DEBUG "status": "success", 23:16:37 DEBUG ----- output ----- 23:16:37 DEBUG "data": { 23:16:37 DEBUG { 23:16:37 DEBUG "resultType": "vector", 23:16:37 DEBUG "status": "success", 23:16:37 DEBUG "result": [ 23:16:37 DEBUG "data": { 23:16:37 DEBUG { 23:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392197" 23:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392197" 23:16:37 DEBUG "resultType": "vector", 23:16:37 DEBUG "metric": { 23:16:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392197" 23:16:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392197" 23:16:37 DEBUG "result": [ 23:16:37 DEBUG }, 23:16:37 INFO [http_cmd]: http status code OK 23:16:37 DEBUG { 23:16:37 DEBUG "value": [ 23:16:37 DEBUG --- status code --- 23:16:37 DEBUG "metric": { 23:16:37 DEBUG 23:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:37 DEBUG "pod": "ds-cts-0" 23:16:37 DEBUG 23:16:37 DEBUG --- http response --- 23:16:37 DEBUG }, 23:16:37 DEBUG 23:16:37 DEBUG "value": [ 23:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:16:37 DEBUG [...] 23:16:37 DEBUG 23:16:37 INFO [http_cmd]: http status code OK 23:16:37 DEBUG ----- output ----- 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG --- status code --- 23:16:37 DEBUG { 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:37 DEBUG "status": "success", 23:16:37 DEBUG 23:16:37 DEBUG [...] 23:16:37 DEBUG --- http response --- 23:16:37 DEBUG "data": { 23:16:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:16:37 DEBUG 23:16:37 INFO [http_cmd]: http status code OK 23:16:37 DEBUG "resultType": "vector", 23:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:37 DEBUG }, 23:16:37 DEBUG 23:16:37 DEBUG --- status code --- 23:16:37 DEBUG "result": [ 23:16:37 DEBUG ----- output ----- 23:16:37 DEBUG "value": [ 23:16:37 DEBUG 23:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:37 DEBUG { 23:16:37 DEBUG { 23:16:37 DEBUG 1713392197, 23:16:37 DEBUG "pod": "ds-cts-2" 23:16:37 DEBUG --- http response --- 23:16:37 DEBUG "metric": { 23:16:37 DEBUG "status": "success", 23:16:37 DEBUG "0.018" 23:16:37 DEBUG }, 23:16:37 DEBUG "pod": "ds-cts-0" 23:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:16:37 DEBUG "data": { 23:16:37 DEBUG ] 23:16:37 DEBUG "value": [ 23:16:37 DEBUG }, 23:16:37 DEBUG ----- output ----- 23:16:37 DEBUG "resultType": "vector", 23:16:37 DEBUG } 23:16:37 DEBUG 1713392197, 23:16:37 DEBUG "value": [ 23:16:37 DEBUG { 23:16:37 DEBUG "result": [ 23:16:37 INFO [http_cmd]: http status code OK 23:16:37 DEBUG ] 23:16:37 DEBUG "0.0004333333333333333" 23:16:37 DEBUG 23:16:37 DEBUG "status": "success", 23:16:37 DEBUG { 23:16:37 DEBUG --- status code --- 23:16:37 DEBUG } 23:16:37 DEBUG ] 23:16:37 DEBUG 23:16:37 DEBUG "data": { 23:16:37 DEBUG "metric": { 23:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:37 DEBUG } 23:16:37 DEBUG } 23:16:37 DEBUG 23:16:37 DEBUG "resultType": "vector", 23:16:37 INFO [http_cmd]: http status code OK 23:16:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:16:37 DEBUG --- http response --- 23:16:37 DEBUG ] 23:16:37 DEBUG [...] 23:16:37 DEBUG "result": [ 23:16:37 DEBUG --- status code --- 23:16:37 DEBUG }, 23:16:37 DEBUG } 23:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:37 DEBUG 23:16:37 DEBUG { 23:16:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:37 DEBUG "value": [ 23:16:37 DEBUG } 23:16:37 DEBUG ----- output ----- 23:16:37 DEBUG 23:16:37 DEBUG "metric": { 23:16:37 DEBUG --- http response --- 23:16:37 DEBUG 23:16:37 DEBUG { 23:16:37 DEBUG 23:16:37 DEBUG "pod": "ds-idrepo-1" 23:16:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:37 DEBUG 23:16:37 DEBUG "status": "success", 23:16:37 DEBUG "pod": "ds-cts-2" 23:16:37 DEBUG }, 23:16:37 DEBUG ----- output ----- 23:16:37 DEBUG 23:16:37 DEBUG "data": { 23:16:37 DEBUG }, 23:16:37 DEBUG "value": [ 23:16:37 DEBUG { 23:16:37 DEBUG [...] 23:16:37 DEBUG "resultType": "vector", 23:16:37 DEBUG "value": [ 23:16:37 DEBUG 23:16:37 DEBUG "status": "success", 23:16:37 DEBUG 23:16:37 DEBUG "result": [ 23:16:37 DEBUG 1713392197, 23:16:37 DEBUG 23:16:37 DEBUG "data": { 23:16:37 DEBUG 23:16:37 DEBUG { 23:16:37 DEBUG "0" 23:16:37 DEBUG 23:16:37 DEBUG "resultType": "vector", 23:16:37 DEBUG 23:16:37 DEBUG "metric": { 23:16:37 DEBUG ] 23:16:37 DEBUG [...] 23:16:37 DEBUG "result": [ 23:16:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:16:37 DEBUG "pod": "ds-idrepo-0" 23:16:37 DEBUG }, 23:16:37 DEBUG 23:16:37 DEBUG { 23:16:37 DEBUG }, 23:16:37 DEBUG } 23:16:37 DEBUG "value": [ 23:16:37 DEBUG 23:16:37 DEBUG "metric": { 23:16:37 DEBUG "value": [ 23:16:37 DEBUG ] 23:16:37 DEBUG 23:16:37 DEBUG 23:16:37 DEBUG "pod": "ds-idrepo-0" 23:16:37 DEBUG 1713392197, 23:16:37 DEBUG } 23:16:37 DEBUG 23:16:37 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:16:37 DEBUG }, 23:16:37 DEBUG "0" 23:16:37 DEBUG } 23:16:37 DEBUG 23:16:37 DEBUG }, 23:16:37 DEBUG "value": [ 23:16:37 DEBUG ] 23:16:37 DEBUG [...] 23:16:37 DEBUG "value": [ 23:16:37 DEBUG 23:16:37 DEBUG } 23:16:37 DEBUG 23:16:37 DEBUG 1713392197, 23:16:37 DEBUG 23:16:37 DEBUG ] 23:16:37 DEBUG 23:16:37 DEBUG "0" 23:16:37 DEBUG 23:16:37 DEBUG } 23:16:37 DEBUG 23:16:37 DEBUG ] 23:16:37 DEBUG [...] 23:16:37 DEBUG } 23:16:37 DEBUG "pod": "ds-idrepo-2" 23:16:37 DEBUG } 23:16:37 DEBUG 23:16:37 DEBUG }, 23:16:37 DEBUG ] 23:16:37 DEBUG 23:16:37 DEBUG "value": [ 23:16:37 DEBUG } 23:16:37 DEBUG 23:16:37 DEBUG 1713392197, 23:16:37 DEBUG } 23:16:37 DEBUG "pod": "ds-idrepo-2" 23:16:37 DEBUG "0" 23:16:37 DEBUG }, 23:16:37 DEBUG ] 23:16:37 DEBUG "value": [ 23:16:37 DEBUG } 23:16:37 DEBUG 1713392197, 23:16:37 DEBUG ] 23:16:37 DEBUG "0" 23:16:37 DEBUG } 23:16:37 DEBUG ] 23:16:37 DEBUG } 23:16:37 DEBUG } 23:16:37 DEBUG ] 23:16:37 DEBUG } 23:16:37 DEBUG } 23:16:37 INFO 23:16:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:16:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:16:37 INFO [loop_until]: OK (rc = 0) 23:16:37 DEBUG --- stdout --- 23:16:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5434Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3772m 12% 6158Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8390m 28% 11129Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3001Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9459m 31% 10282Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 4928Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8564m 28% 10006Mi 8% gke-xlou-cdm-ds-0f601751-cf7b 19047m 63% 3647Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 8598m 28% 9588Mi 8% gke-xlou-cdm-ds-0f601751-j488 4140m 13% 5520Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 9485m 31% 11380Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 3872m 12% 5999Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 10652m 35% 9730Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 8157m 51% 8155Mi 13% 23:16:37 DEBUG --- stderr --- 23:16:37 DEBUG 23:17:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392254" 23:17:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713392254" 23:17:34 INFO [http_cmd]: http status code OK 23:17:34 DEBUG --- status code --- 23:17:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:34 DEBUG --- http response --- 23:17:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:17:34 ERROR [http_cmd]: ERROR 23:17:34 ERROR --- status code --- 23:17:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:17:34 ERROR --- http response --- 23:17:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:17:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:17:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392254" 23:17:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392254" 23:17:34 INFO [http_cmd]: http status code OK 23:17:34 DEBUG --- status code --- 23:17:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:34 DEBUG --- http response --- 23:17:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:34 DEBUG ----- output ----- 23:17:34 DEBUG { 23:17:34 DEBUG "status": "success", 23:17:34 DEBUG "data": { 23:17:34 DEBUG "resultType": "vector", 23:17:34 DEBUG "result": [ 23:17:34 DEBUG { 23:17:34 DEBUG "metric": { 23:17:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:17:34 DEBUG }, 23:17:34 DEBUG "value": [ 23:17:34 DEBUG 23:17:34 DEBUG 23:17:34 DEBUG 23:17:34 DEBUG [...] 23:17:34 DEBUG 23:17:34 DEBUG 23:17:34 DEBUG 23:17:34 DEBUG "pod": "am-b59c765b7-bgsxp" 23:17:34 DEBUG }, 23:17:34 DEBUG "value": [ 23:17:34 DEBUG 1713392254, 23:17:34 DEBUG "1090.9333333333334" 23:17:34 DEBUG ] 23:17:34 DEBUG } 23:17:34 DEBUG ] 23:17:34 DEBUG } 23:17:34 DEBUG } 23:17:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392254" 23:17:34 INFO [http_cmd]: http status code OK 23:17:34 DEBUG --- status code --- 23:17:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:34 DEBUG --- http response --- 23:17:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:17:34 DEBUG ----- output ----- 23:17:34 DEBUG { 23:17:34 DEBUG "status": "success", 23:17:34 DEBUG "data": { 23:17:34 DEBUG "resultType": "vector", 23:17:34 DEBUG "result": [ 23:17:34 DEBUG { 23:17:34 DEBUG "metric": { 23:17:34 DEBUG "pod": "ds-idrepo-1" 23:17:34 DEBUG }, 23:17:34 DEBUG "value": [ 23:17:34 DEBUG 23:17:34 DEBUG 23:17:34 DEBUG 23:17:34 DEBUG [...] 23:17:34 DEBUG 23:17:34 DEBUG 23:17:34 DEBUG 23:17:34 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 23:17:34 DEBUG }, 23:17:34 DEBUG "value": [ 23:17:34 DEBUG 1713392254, 23:17:34 DEBUG "0.000019020211357851533" 23:17:34 DEBUG ] 23:17:34 DEBUG } 23:17:34 DEBUG ] 23:17:34 DEBUG } 23:17:34 DEBUG } 23:17:34 INFO [http_cmd]: http status code OK 23:17:34 DEBUG --- status code --- 23:17:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:34 DEBUG --- http response --- 23:17:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:34 DEBUG ----- output ----- 23:17:34 DEBUG { 23:17:34 DEBUG "status": "success", 23:17:34 DEBUG "data": { 23:17:34 DEBUG "resultType": "vector", 23:17:34 DEBUG "result": [ 23:17:34 DEBUG { 23:17:34 DEBUG "metric": { 23:17:34 DEBUG "instance": "10.142.0.107:9100" 23:17:34 DEBUG }, 23:17:34 DEBUG "value": [ 23:17:34 DEBUG 23:17:34 DEBUG 23:17:34 DEBUG 23:17:34 DEBUG [...] 23:17:34 DEBUG 23:17:34 DEBUG 23:17:34 DEBUG 23:17:34 DEBUG "instance": "10.142.0.43:9100" 23:17:34 DEBUG }, 23:17:34 DEBUG "value": [ 23:17:34 DEBUG 1713392254, 23:17:34 DEBUG "0.04616666666666636" 23:17:34 DEBUG ] 23:17:34 DEBUG } 23:17:34 DEBUG ] 23:17:34 DEBUG } 23:17:34 DEBUG } 23:17:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392254" 23:17:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713392254" 23:17:35 INFO [http_cmd]: http status code OK 23:17:35 DEBUG --- status code --- 23:17:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:35 DEBUG --- http response --- 23:17:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:35 DEBUG ----- output ----- 23:17:35 DEBUG { 23:17:35 DEBUG "status": "success", 23:17:35 DEBUG "data": { 23:17:35 DEBUG "resultType": "vector", 23:17:35 DEBUG "result": [ 23:17:35 DEBUG { 23:17:35 DEBUG "metric": { 23:17:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:17:35 DEBUG }, 23:17:35 DEBUG "value": [ 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG [...] 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:17:35 DEBUG }, 23:17:35 DEBUG "value": [ 23:17:35 DEBUG 1713392254, 23:17:35 DEBUG "5803970560" 23:17:35 DEBUG ] 23:17:35 DEBUG } 23:17:35 DEBUG ] 23:17:35 DEBUG } 23:17:35 DEBUG } 23:17:35 INFO [http_cmd]: http status code OK 23:17:35 DEBUG --- status code --- 23:17:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:35 DEBUG --- http response --- 23:17:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:35 DEBUG ----- output ----- 23:17:35 DEBUG { 23:17:35 DEBUG "status": "success", 23:17:35 DEBUG "data": { 23:17:35 DEBUG "resultType": "vector", 23:17:35 DEBUG "result": [ 23:17:35 DEBUG { 23:17:35 DEBUG "metric": { 23:17:35 DEBUG "instance": "10.142.0.107:9100" 23:17:35 DEBUG }, 23:17:35 DEBUG "value": [ 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG [...] 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG "instance": "10.142.0.43:9100" 23:17:35 DEBUG }, 23:17:35 DEBUG "value": [ 23:17:35 DEBUG 1713392254, 23:17:35 DEBUG "41201935.166666664" 23:17:35 DEBUG ] 23:17:35 DEBUG } 23:17:35 DEBUG ] 23:17:35 DEBUG } 23:17:35 DEBUG } 23:17:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392255" 23:17:35 INFO [http_cmd]: http status code OK 23:17:35 DEBUG --- status code --- 23:17:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:35 DEBUG --- http response --- 23:17:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:17:35 DEBUG ----- output ----- 23:17:35 DEBUG { 23:17:35 DEBUG "status": "success", 23:17:35 DEBUG "data": { 23:17:35 DEBUG "resultType": "vector", 23:17:35 DEBUG "result": [ 23:17:35 DEBUG { 23:17:35 DEBUG "metric": { 23:17:35 DEBUG "pod": "ds-cts-0" 23:17:35 DEBUG }, 23:17:35 DEBUG "value": [ 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG [...] 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG "pod": "ds-cts-2" 23:17:35 DEBUG }, 23:17:35 DEBUG "value": [ 23:17:35 DEBUG 1713392255, 23:17:35 DEBUG "29.933333333333334" 23:17:35 DEBUG ] 23:17:35 DEBUG } 23:17:35 DEBUG ] 23:17:35 DEBUG } 23:17:35 DEBUG } 23:17:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392255" 23:17:35 INFO [http_cmd]: http status code OK 23:17:35 DEBUG --- status code --- 23:17:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:35 DEBUG --- http response --- 23:17:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:35 DEBUG ----- output ----- 23:17:35 DEBUG { 23:17:35 DEBUG "status": "success", 23:17:35 DEBUG "data": { 23:17:35 DEBUG "resultType": "vector", 23:17:35 DEBUG "result": [ 23:17:35 DEBUG { 23:17:35 DEBUG "metric": { 23:17:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:17:35 DEBUG }, 23:17:35 DEBUG "value": [ 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG [...] 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG 23:17:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:17:35 DEBUG }, 23:17:35 DEBUG "value": [ 23:17:35 DEBUG 1713392255, 23:17:35 DEBUG "1822.6666666666667" 23:17:35 DEBUG ] 23:17:35 DEBUG } 23:17:35 DEBUG ] 23:17:35 DEBUG } 23:17:35 DEBUG } 23:17:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392256" 23:17:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392256" 23:17:36 INFO [http_cmd]: http status code OK 23:17:36 DEBUG --- status code --- 23:17:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:36 DEBUG --- http response --- 23:17:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:36 DEBUG ----- output ----- 23:17:36 DEBUG { 23:17:36 DEBUG "status": "success", 23:17:36 DEBUG "data": { 23:17:36 DEBUG "resultType": "vector", 23:17:36 DEBUG "result": [ 23:17:36 DEBUG { 23:17:36 DEBUG "metric": { 23:17:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:17:36 DEBUG }, 23:17:36 DEBUG "value": [ 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG [...] 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:17:36 DEBUG }, 23:17:36 DEBUG "value": [ 23:17:36 DEBUG 1713392256, 23:17:36 DEBUG "1090.8666666666666" 23:17:36 DEBUG ] 23:17:36 DEBUG } 23:17:36 DEBUG ] 23:17:36 DEBUG } 23:17:36 DEBUG } 23:17:36 INFO [http_cmd]: http status code OK 23:17:36 DEBUG --- status code --- 23:17:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:36 DEBUG --- http response --- 23:17:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:17:36 DEBUG ----- output ----- 23:17:36 DEBUG { 23:17:36 DEBUG "status": "success", 23:17:36 DEBUG "data": { 23:17:36 DEBUG "resultType": "vector", 23:17:36 DEBUG "result": [ 23:17:36 DEBUG { 23:17:36 DEBUG "metric": { 23:17:36 DEBUG "pod": "ds-idrepo-1" 23:17:36 DEBUG }, 23:17:36 DEBUG "value": [ 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG [...] 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 23:17:36 DEBUG }, 23:17:36 DEBUG "value": [ 23:17:36 DEBUG 1713392256, 23:17:36 DEBUG "104.2302055406613" 23:17:36 DEBUG ] 23:17:36 DEBUG } 23:17:36 DEBUG ] 23:17:36 DEBUG } 23:17:36 DEBUG } 23:17:36 INFO 23:17:36 INFO [loop_until]: kubectl --namespace=xlou top pods 23:17:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:17:36 INFO [loop_until]: OK (rc = 0) 23:17:36 DEBUG --- stdout --- 23:17:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 7910m 7391Mi am-b59c765b7-bgsxp 7151m 7758Mi am-b59c765b7-ch5q4 7763m 8543Mi ds-cts-0 9038m 8018Mi ds-cts-1 8501m 7762Mi ds-cts-2 9382m 7996Mi ds-idrepo-0 3321m 3223Mi ds-idrepo-1 3382m 2880Mi ds-idrepo-2 3298m 2935Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 967Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8107m 6840Mi 23:17:36 DEBUG --- stderr --- 23:17:36 DEBUG 23:17:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392256" 23:17:36 INFO [http_cmd]: http status code OK 23:17:36 DEBUG --- status code --- 23:17:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:36 DEBUG --- http response --- 23:17:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:36 DEBUG ----- output ----- 23:17:36 DEBUG { 23:17:36 DEBUG "status": "success", 23:17:36 DEBUG "data": { 23:17:36 DEBUG "resultType": "vector", 23:17:36 DEBUG "result": [ 23:17:36 DEBUG { 23:17:36 DEBUG "metric": { 23:17:36 DEBUG "instance": "10.142.0.107:9100" 23:17:36 DEBUG }, 23:17:36 DEBUG "value": [ 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG [...] 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG "instance": "10.142.0.43:9100" 23:17:36 DEBUG }, 23:17:36 DEBUG "value": [ 23:17:36 DEBUG 1713392256, 23:17:36 DEBUG "249463272448" 23:17:36 DEBUG ] 23:17:36 DEBUG } 23:17:36 DEBUG ] 23:17:36 DEBUG } 23:17:36 DEBUG } 23:17:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392256" 23:17:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392256" 23:17:36 INFO [http_cmd]: http status code OK 23:17:36 INFO [http_cmd]: http status code OK 23:17:36 DEBUG --- status code --- 23:17:36 DEBUG --- status code --- 23:17:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:36 DEBUG --- http response --- 23:17:36 DEBUG --- http response --- 23:17:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:17:36 DEBUG ----- output ----- 23:17:36 DEBUG { 23:17:36 DEBUG "status": "success", 23:17:36 DEBUG "data": { 23:17:36 DEBUG "resultType": "vector", 23:17:36 DEBUG "result": [ 23:17:36 DEBUG { 23:17:36 DEBUG "metric": { 23:17:36 DEBUG "pod": "ds-idrepo-1" 23:17:36 DEBUG }, 23:17:36 DEBUG "value": [ 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG [...] 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 23:17:36 DEBUG }, 23:17:36 DEBUG "value": [ 23:17:36 DEBUG 1713392256, 23:17:36 DEBUG "0" 23:17:36 DEBUG ] 23:17:36 DEBUG } 23:17:36 DEBUG ] 23:17:36 DEBUG } 23:17:36 DEBUG } 23:17:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:36 DEBUG ----- output ----- 23:17:36 DEBUG { 23:17:36 DEBUG "status": "success", 23:17:36 DEBUG "data": { 23:17:36 DEBUG "resultType": "vector", 23:17:36 DEBUG "result": [ 23:17:36 DEBUG { 23:17:36 DEBUG "metric": { 23:17:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:17:36 DEBUG }, 23:17:36 DEBUG "value": [ 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG [...] 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:17:36 DEBUG }, 23:17:36 DEBUG "value": [ 23:17:36 DEBUG 1713392256, 23:17:36 DEBUG "1090.7666666666667" 23:17:36 DEBUG ] 23:17:36 DEBUG } 23:17:36 DEBUG ] 23:17:36 DEBUG } 23:17:36 DEBUG } 23:17:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713392256" 23:17:36 INFO [http_cmd]: http status code OK 23:17:36 DEBUG --- status code --- 23:17:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:36 DEBUG --- http response --- 23:17:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:36 DEBUG ----- output ----- 23:17:36 DEBUG { 23:17:36 DEBUG "status": "success", 23:17:36 DEBUG "data": { 23:17:36 DEBUG "resultType": "vector", 23:17:36 DEBUG "result": [ 23:17:36 DEBUG { 23:17:36 DEBUG "metric": { 23:17:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:17:36 DEBUG }, 23:17:36 DEBUG "value": [ 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG [...] 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG 23:17:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:17:36 DEBUG }, 23:17:36 DEBUG "value": [ 23:17:36 DEBUG 1713392256, 23:17:36 DEBUG "8.029830532369356" 23:17:36 DEBUG ] 23:17:36 DEBUG } 23:17:36 DEBUG ] 23:17:36 DEBUG } 23:17:36 DEBUG } 23:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713392257" 23:17:37 INFO [http_cmd]: http status code OK 23:17:37 DEBUG --- status code --- 23:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:37 DEBUG --- http response --- 23:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:17:37 DEBUG ----- output ----- 23:17:37 DEBUG { 23:17:37 DEBUG "status": "success", 23:17:37 DEBUG "data": { 23:17:37 DEBUG "resultType": "vector", 23:17:37 DEBUG "result": [ 23:17:37 DEBUG { 23:17:37 DEBUG "metric": { 23:17:37 DEBUG "pod": "ds-idrepo-0" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG [...] 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG "pod": "ds-cts-2" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 1713392257, 23:17:37 DEBUG "0" 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG } 23:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392257" 23:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392257" 23:17:37 INFO [http_cmd]: http status code OK 23:17:37 DEBUG --- status code --- 23:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:37 DEBUG --- http response --- 23:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:37 DEBUG ----- output ----- 23:17:37 DEBUG { 23:17:37 DEBUG "status": "success", 23:17:37 DEBUG "data": { 23:17:37 DEBUG "resultType": "vector", 23:17:37 DEBUG "result": [ 23:17:37 DEBUG { 23:17:37 DEBUG "metric": { 23:17:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG [...] 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 1713392257, 23:17:37 DEBUG "179019" 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG } 23:17:37 INFO [http_cmd]: http status code OK 23:17:37 DEBUG --- status code --- 23:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:37 DEBUG --- http response --- 23:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:37 DEBUG ----- output ----- 23:17:37 DEBUG { 23:17:37 DEBUG "status": "success", 23:17:37 DEBUG "data": { 23:17:37 DEBUG "resultType": "vector", 23:17:37 DEBUG "result": [ 23:17:37 DEBUG { 23:17:37 DEBUG "metric": { 23:17:37 DEBUG "instance": "10.142.0.107:9100" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG [...] 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG "instance": "10.142.0.43:9100" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 1713392257, 23:17:37 DEBUG "0.03166666666666667" 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG } 23:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392257" 23:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392257" 23:17:37 INFO [http_cmd]: http status code OK 23:17:37 DEBUG --- status code --- 23:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:37 DEBUG --- http response --- 23:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:37 DEBUG ----- output ----- 23:17:37 DEBUG { 23:17:37 DEBUG "status": "success", 23:17:37 DEBUG "data": { 23:17:37 DEBUG "resultType": "vector", 23:17:37 DEBUG "result": [ 23:17:37 DEBUG { 23:17:37 DEBUG "metric": { 23:17:37 DEBUG "pod": "ds-idrepo-0" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG [...] 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG "pod": "ds-idrepo-2" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 1713392257, 23:17:37 DEBUG "0" 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG } 23:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392257" 23:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392257" 23:17:37 INFO [http_cmd]: http status code OK 23:17:37 DEBUG --- status code --- 23:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:37 DEBUG --- http response --- 23:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:37 DEBUG ----- output ----- 23:17:37 DEBUG { 23:17:37 DEBUG "status": "success", 23:17:37 DEBUG "data": { 23:17:37 DEBUG "resultType": "vector", 23:17:37 DEBUG "result": [ 23:17:37 DEBUG { 23:17:37 DEBUG "metric": { 23:17:37 DEBUG "instance": "10.142.0.107:9100" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG [...] 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG "instance": "10.142.0.43:9100" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 1713392257, 23:17:37 DEBUG "41199973.6" 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG } 23:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713392257" 23:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392257" 23:17:37 INFO [http_cmd]: http status code OK 23:17:37 DEBUG --- status code --- 23:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:37 DEBUG --- http response --- 23:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:37 DEBUG ----- output ----- 23:17:37 DEBUG { 23:17:37 DEBUG "status": "success", 23:17:37 DEBUG "data": { 23:17:37 DEBUG "resultType": "vector", 23:17:37 DEBUG "result": [ 23:17:37 DEBUG { 23:17:37 DEBUG "metric": { 23:17:37 DEBUG "pod": "ds-idrepo-0" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG [...] 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG "pod": "ds-idrepo-2" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 1713392257, 23:17:37 DEBUG "0" 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG } 23:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392257" 23:17:37 INFO [http_cmd]: http status code OK 23:17:37 DEBUG --- status code --- 23:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:37 DEBUG --- http response --- 23:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:37 DEBUG ----- output ----- 23:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392257" 23:17:37 DEBUG { 23:17:37 DEBUG "status": "success", 23:17:37 DEBUG "data": { 23:17:37 DEBUG "resultType": "vector", 23:17:37 DEBUG "result": [ 23:17:37 DEBUG { 23:17:37 DEBUG "metric": { 23:17:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG [...] 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 1713392257, 23:17:37 DEBUG "0" 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG ] 23:17:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392257" 23:17:37 DEBUG } 23:17:37 DEBUG } 23:17:37 INFO [http_cmd]: http status code OK 23:17:37 DEBUG --- status code --- 23:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:37 DEBUG --- http response --- 23:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:37 DEBUG ----- output ----- 23:17:37 DEBUG { 23:17:37 DEBUG "status": "success", 23:17:37 DEBUG "data": { 23:17:37 DEBUG "resultType": "vector", 23:17:37 DEBUG "result": [ 23:17:37 DEBUG { 23:17:37 DEBUG "metric": { 23:17:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG [...] 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 1713392257, 23:17:37 DEBUG "0.018" 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG } 23:17:37 INFO [http_cmd]: http status code OK 23:17:37 DEBUG --- status code --- 23:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:37 DEBUG --- http response --- 23:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:17:37 DEBUG ----- output ----- 23:17:37 DEBUG { 23:17:37 DEBUG "status": "success", 23:17:37 DEBUG "data": { 23:17:37 DEBUG "resultType": "vector", 23:17:37 DEBUG "result": [ 23:17:37 DEBUG { 23:17:37 DEBUG "metric": { 23:17:37 DEBUG "pod": "ds-cts-0" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG [...] 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG "pod": "ds-cts-2" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 1713392257, 23:17:37 DEBUG "0.0013420066666666665" 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG } 23:17:37 INFO [http_cmd]: http status code OK 23:17:37 DEBUG --- status code --- 23:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:37 DEBUG --- http response --- 23:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:17:37 DEBUG ----- output ----- 23:17:37 DEBUG { 23:17:37 DEBUG "status": "success", 23:17:37 DEBUG "data": { 23:17:37 DEBUG "resultType": "vector", 23:17:37 DEBUG "result": [ 23:17:37 DEBUG { 23:17:37 DEBUG "metric": { 23:17:37 INFO [http_cmd]: http status code OK 23:17:37 DEBUG "pod": "ds-cts-0" 23:17:37 DEBUG --- status code --- 23:17:37 DEBUG }, 23:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:37 DEBUG "value": [ 23:17:37 DEBUG --- http response --- 23:17:37 DEBUG 23:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 23:17:37 DEBUG 23:17:37 DEBUG ----- output ----- 23:17:37 DEBUG 23:17:37 DEBUG { 23:17:37 DEBUG [...] 23:17:37 DEBUG "status": "success", 23:17:37 DEBUG 23:17:37 DEBUG "data": { 23:17:37 DEBUG 23:17:37 DEBUG "resultType": "vector", 23:17:37 DEBUG 23:17:37 DEBUG "result": [ 23:17:37 DEBUG { 23:17:37 DEBUG "metric": { 23:17:37 DEBUG "pod": "ds-cts-2" 23:17:37 DEBUG "pod": "ds-idrepo-0" 23:17:37 DEBUG }, 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 1713392257, 23:17:37 DEBUG 23:17:37 DEBUG "0" 23:17:37 DEBUG 23:17:37 DEBUG ] 23:17:37 DEBUG 23:17:37 DEBUG } 23:17:37 DEBUG [...] 23:17:37 DEBUG ] 23:17:37 DEBUG 23:17:37 DEBUG } 23:17:37 DEBUG 23:17:37 DEBUG } 23:17:37 DEBUG 23:17:37 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 1713392257, 23:17:37 DEBUG "0" 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG } 23:17:37 INFO [http_cmd]: http status code OK 23:17:37 DEBUG --- status code --- 23:17:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:37 DEBUG --- http response --- 23:17:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:37 DEBUG ----- output ----- 23:17:37 DEBUG { 23:17:37 DEBUG "status": "success", 23:17:37 DEBUG "data": { 23:17:37 DEBUG "resultType": "vector", 23:17:37 DEBUG "result": [ 23:17:37 DEBUG { 23:17:37 DEBUG "metric": { 23:17:37 DEBUG "pod": "ds-idrepo-0" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG [...] 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG 23:17:37 DEBUG "pod": "ds-idrepo-2" 23:17:37 DEBUG }, 23:17:37 DEBUG "value": [ 23:17:37 DEBUG 1713392257, 23:17:37 DEBUG "0" 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG ] 23:17:37 DEBUG } 23:17:37 DEBUG } 23:17:37 INFO 23:17:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:17:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:17:37 INFO [loop_until]: OK (rc = 0) 23:17:37 DEBUG --- stdout --- 23:17:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 5426Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3442m 11% 6161Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7673m 25% 11127Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3001Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9014m 30% 10284Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 4952Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 7803m 26% 10007Mi 8% gke-xlou-cdm-ds-0f601751-cf7b 16200m 54% 3687Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 7660m 25% 9602Mi 8% gke-xlou-cdm-ds-0f601751-j488 3596m 12% 5517Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 8453m 28% 11377Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 3522m 11% 6001Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 9364m 31% 9738Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 7926m 49% 10171Mi 17% 23:17:37 DEBUG --- stderr --- 23:17:37 DEBUG 23:18:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392314" 23:18:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713392314" 23:18:34 ERROR [http_cmd]: ERROR 23:18:34 ERROR --- status code --- 23:18:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:18:34 ERROR --- http response --- 23:18:34 INFO [http_cmd]: http status code OK 23:18:34 DEBUG --- status code --- 23:18:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:18:34 DEBUG --- http response --- 23:18:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:18:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:18:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392314" 23:18:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392314" 23:18:34 INFO [http_cmd]: http status code OK 23:18:34 DEBUG --- status code --- 23:18:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:34 DEBUG --- http response --- 23:18:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:34 DEBUG ----- output ----- 23:18:34 DEBUG { 23:18:34 DEBUG "status": "success", 23:18:34 DEBUG "data": { 23:18:34 DEBUG "resultType": "vector", 23:18:34 DEBUG "result": [ 23:18:34 DEBUG { 23:18:34 DEBUG "metric": { 23:18:34 DEBUG "pod": "am-b59c765b7-ch5q4" 23:18:34 DEBUG }, 23:18:34 DEBUG "value": [ 23:18:34 DEBUG 23:18:34 DEBUG 23:18:34 DEBUG 23:18:34 DEBUG [...] 23:18:34 DEBUG 23:18:34 DEBUG 23:18:34 DEBUG 23:18:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:18:35 DEBUG }, 23:18:35 DEBUG "value": [ 23:18:35 DEBUG 1713392314, 23:18:35 DEBUG "1047.9666666666667" 23:18:35 DEBUG ] 23:18:35 DEBUG } 23:18:35 DEBUG ] 23:18:35 DEBUG } 23:18:35 DEBUG } 23:18:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392315" 23:18:35 INFO [http_cmd]: http status code OK 23:18:35 DEBUG --- status code --- 23:18:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:35 DEBUG --- http response --- 23:18:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:18:35 DEBUG ----- output ----- 23:18:35 DEBUG { 23:18:35 DEBUG "status": "success", 23:18:35 DEBUG "data": { 23:18:35 DEBUG "resultType": "vector", 23:18:35 DEBUG "result": [ 23:18:35 DEBUG { 23:18:35 DEBUG "metric": { 23:18:35 DEBUG "pod": "ds-idrepo-1" 23:18:35 DEBUG }, 23:18:35 DEBUG "value": [ 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG [...] 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:18:35 DEBUG }, 23:18:35 DEBUG "value": [ 23:18:35 DEBUG 1713392314, 23:18:35 DEBUG "0" 23:18:35 DEBUG ] 23:18:35 DEBUG } 23:18:35 DEBUG ] 23:18:35 DEBUG } 23:18:35 DEBUG } 23:18:35 INFO [http_cmd]: http status code OK 23:18:35 DEBUG --- status code --- 23:18:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:35 DEBUG --- http response --- 23:18:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:35 DEBUG ----- output ----- 23:18:35 DEBUG { 23:18:35 DEBUG "status": "success", 23:18:35 DEBUG "data": { 23:18:35 DEBUG "resultType": "vector", 23:18:35 DEBUG "result": [ 23:18:35 DEBUG { 23:18:35 DEBUG "metric": { 23:18:35 DEBUG "instance": "10.142.0.107:9100" 23:18:35 DEBUG }, 23:18:35 DEBUG "value": [ 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG [...] 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG "instance": "10.142.0.43:9100" 23:18:35 DEBUG }, 23:18:35 DEBUG "value": [ 23:18:35 DEBUG 1713392315, 23:18:35 DEBUG "0.02866666666666807" 23:18:35 DEBUG ] 23:18:35 DEBUG } 23:18:35 DEBUG ] 23:18:35 DEBUG } 23:18:35 DEBUG } 23:18:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713392315" 23:18:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392315" 23:18:35 INFO [http_cmd]: http status code OK 23:18:35 DEBUG --- status code --- 23:18:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:35 DEBUG --- http response --- 23:18:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:35 DEBUG ----- output ----- 23:18:35 DEBUG { 23:18:35 DEBUG "status": "success", 23:18:35 INFO [http_cmd]: http status code OK 23:18:35 DEBUG "data": { 23:18:35 DEBUG --- status code --- 23:18:35 DEBUG "resultType": "vector", 23:18:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:35 DEBUG "result": [ 23:18:35 DEBUG --- http response --- 23:18:35 DEBUG { 23:18:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:35 DEBUG "metric": { 23:18:35 DEBUG ----- output ----- 23:18:35 DEBUG "instance": "10.142.0.107:9100" 23:18:35 DEBUG { 23:18:35 DEBUG }, 23:18:35 DEBUG "status": "success", 23:18:35 DEBUG "value": [ 23:18:35 DEBUG "data": { 23:18:35 DEBUG 23:18:35 DEBUG "resultType": "vector", 23:18:35 DEBUG 23:18:35 DEBUG "result": [ 23:18:35 DEBUG 23:18:35 DEBUG { 23:18:35 DEBUG [...] 23:18:35 DEBUG "metric": { 23:18:35 DEBUG 23:18:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:18:35 DEBUG 23:18:35 DEBUG }, 23:18:35 DEBUG 23:18:35 DEBUG "value": [ 23:18:35 DEBUG "instance": "10.142.0.43:9100" 23:18:35 DEBUG 23:18:35 DEBUG }, 23:18:35 DEBUG 23:18:35 DEBUG "value": [ 23:18:35 DEBUG 23:18:35 DEBUG 1713392315, 23:18:35 DEBUG [...] 23:18:35 DEBUG "27284518.166666664" 23:18:35 DEBUG 23:18:35 DEBUG ] 23:18:35 DEBUG 23:18:35 DEBUG } 23:18:35 DEBUG 23:18:35 DEBUG ] 23:18:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:18:35 DEBUG } 23:18:35 DEBUG }, 23:18:35 DEBUG } 23:18:35 DEBUG "value": [ 23:18:35 DEBUG 1713392315, 23:18:35 DEBUG "7507750912" 23:18:35 DEBUG ] 23:18:35 DEBUG } 23:18:35 DEBUG ] 23:18:35 DEBUG } 23:18:35 DEBUG } 23:18:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392315" 23:18:35 INFO [http_cmd]: http status code OK 23:18:35 DEBUG --- status code --- 23:18:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:35 DEBUG --- http response --- 23:18:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:18:35 DEBUG ----- output ----- 23:18:35 DEBUG { 23:18:35 DEBUG "status": "success", 23:18:35 DEBUG "data": { 23:18:35 DEBUG "resultType": "vector", 23:18:35 DEBUG "result": [ 23:18:35 DEBUG { 23:18:35 DEBUG "metric": { 23:18:35 DEBUG "pod": "ds-cts-0" 23:18:35 DEBUG }, 23:18:35 DEBUG "value": [ 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG [...] 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG "pod": "ds-cts-2" 23:18:35 DEBUG }, 23:18:35 DEBUG "value": [ 23:18:35 DEBUG 1713392315, 23:18:35 DEBUG "29.933333333333334" 23:18:35 DEBUG ] 23:18:35 DEBUG } 23:18:35 DEBUG ] 23:18:35 DEBUG } 23:18:35 DEBUG } 23:18:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392315" 23:18:35 INFO [http_cmd]: http status code OK 23:18:35 DEBUG --- status code --- 23:18:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:35 DEBUG --- http response --- 23:18:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:35 DEBUG ----- output ----- 23:18:35 DEBUG { 23:18:35 DEBUG "status": "success", 23:18:35 DEBUG "data": { 23:18:35 DEBUG "resultType": "vector", 23:18:35 DEBUG "result": [ 23:18:35 DEBUG { 23:18:35 DEBUG "metric": { 23:18:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:18:35 DEBUG }, 23:18:35 DEBUG "value": [ 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG [...] 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG 23:18:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:18:35 DEBUG }, 23:18:35 DEBUG "value": [ 23:18:35 DEBUG 1713392315, 23:18:35 DEBUG "102.06666666666666" 23:18:35 DEBUG ] 23:18:35 DEBUG } 23:18:35 DEBUG ] 23:18:35 DEBUG } 23:18:35 DEBUG } 23:18:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392316" 23:18:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392316" 23:18:36 INFO [http_cmd]: http status code OK 23:18:36 DEBUG --- status code --- 23:18:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:36 DEBUG --- http response --- 23:18:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:36 DEBUG ----- output ----- 23:18:36 DEBUG { 23:18:36 DEBUG "status": "success", 23:18:36 DEBUG "data": { 23:18:36 DEBUG "resultType": "vector", 23:18:36 DEBUG "result": [ 23:18:36 DEBUG { 23:18:36 DEBUG "metric": { 23:18:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:18:36 DEBUG }, 23:18:36 DEBUG "value": [ 23:18:36 DEBUG 23:18:36 DEBUG 23:18:36 DEBUG 23:18:36 DEBUG [...] 23:18:36 DEBUG 23:18:36 DEBUG 23:18:36 DEBUG 23:18:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:18:36 DEBUG }, 23:18:36 DEBUG "value": [ 23:18:36 DEBUG 1713392316, 23:18:36 DEBUG "1047.9666666666667" 23:18:36 DEBUG ] 23:18:36 DEBUG } 23:18:36 DEBUG ] 23:18:36 DEBUG } 23:18:36 DEBUG } 23:18:36 INFO [http_cmd]: http status code OK 23:18:36 DEBUG --- status code --- 23:18:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:36 DEBUG --- http response --- 23:18:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:18:36 DEBUG ----- output ----- 23:18:36 DEBUG { 23:18:36 DEBUG "status": "success", 23:18:36 DEBUG "data": { 23:18:36 DEBUG "resultType": "vector", 23:18:36 DEBUG "result": [ 23:18:36 DEBUG { 23:18:36 DEBUG "metric": { 23:18:36 DEBUG "pod": "ds-idrepo-1" 23:18:36 DEBUG }, 23:18:36 DEBUG "value": [ 23:18:36 DEBUG 23:18:36 DEBUG 23:18:36 DEBUG 23:18:36 DEBUG [...] 23:18:36 DEBUG 23:18:36 DEBUG 23:18:36 DEBUG 23:18:36 DEBUG "pod": "ds-idrepo-2" 23:18:36 DEBUG }, 23:18:36 DEBUG "value": [ 23:18:36 DEBUG 1713392316, 23:18:36 DEBUG "2747.5747200424307" 23:18:36 DEBUG ] 23:18:36 DEBUG } 23:18:36 DEBUG ] 23:18:36 DEBUG } 23:18:36 DEBUG } 23:18:36 INFO 23:18:36 INFO [loop_until]: kubectl --namespace=xlou top pods 23:18:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:18:36 INFO [loop_until]: OK (rc = 0) 23:18:36 DEBUG --- stdout --- 23:18:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 5959m 7388Mi am-b59c765b7-bgsxp 4922m 7752Mi am-b59c765b7-ch5q4 5542m 8542Mi ds-cts-0 3582m 8004Mi ds-cts-1 3338m 7794Mi ds-cts-2 3648m 8034Mi ds-idrepo-0 2776m 3222Mi ds-idrepo-1 2787m 2880Mi ds-idrepo-2 2722m 2935Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 12m 970Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7890m 7845Mi 23:18:36 DEBUG --- stderr --- 23:18:36 DEBUG 23:18:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392316" 23:18:36 INFO [http_cmd]: http status code OK 23:18:36 DEBUG --- status code --- 23:18:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:36 DEBUG --- http response --- 23:18:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:36 DEBUG ----- output ----- 23:18:36 DEBUG { 23:18:36 DEBUG "status": "success", 23:18:36 DEBUG "data": { 23:18:36 DEBUG "resultType": "vector", 23:18:36 DEBUG "result": [ 23:18:36 DEBUG { 23:18:36 DEBUG "metric": { 23:18:36 DEBUG "instance": "10.142.0.107:9100" 23:18:36 DEBUG }, 23:18:36 DEBUG "value": [ 23:18:36 DEBUG 23:18:36 DEBUG 23:18:36 DEBUG 23:18:36 DEBUG [...] 23:18:36 DEBUG 23:18:36 DEBUG 23:18:36 DEBUG 23:18:36 DEBUG "instance": "10.142.0.43:9100" 23:18:36 DEBUG }, 23:18:36 DEBUG "value": [ 23:18:36 DEBUG 1713392316, 23:18:36 DEBUG "251074391040" 23:18:36 DEBUG ] 23:18:36 DEBUG } 23:18:36 DEBUG ] 23:18:36 DEBUG } 23:18:36 DEBUG } 23:18:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392316" 23:18:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392316" 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG --- status code --- 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 DEBUG --- http response --- 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:37 DEBUG --- status code --- 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 DEBUG { 23:18:37 DEBUG --- http response --- 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:18:37 DEBUG "data": { 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG { 23:18:37 DEBUG "result": [ 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG { 23:18:37 DEBUG "data": { 23:18:37 DEBUG "metric": { 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:18:37 DEBUG "result": [ 23:18:37 DEBUG }, 23:18:37 DEBUG { 23:18:37 DEBUG "value": [ 23:18:37 DEBUG "metric": { 23:18:37 DEBUG 23:18:37 DEBUG "pod": "ds-idrepo-1" 23:18:37 DEBUG 23:18:37 DEBUG }, 23:18:37 DEBUG 23:18:37 DEBUG "value": [ 23:18:37 DEBUG [...] 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG [...] 23:18:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:18:37 DEBUG 23:18:37 DEBUG }, 23:18:37 DEBUG 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 23:18:37 DEBUG 1713392316, 23:18:37 DEBUG "pod": "ds-idrepo-2" 23:18:37 DEBUG "1047.9666666666667" 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG ] 23:18:37 DEBUG }, 23:18:37 DEBUG } 23:18:37 DEBUG "value": [ 23:18:37 DEBUG } 23:18:37 DEBUG 1713392316, 23:18:37 DEBUG "0" 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713392317" 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG --- status code --- 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 DEBUG --- http response --- 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG { 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG "data": { 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG "result": [ 23:18:37 DEBUG { 23:18:37 DEBUG "metric": { 23:18:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:18:37 DEBUG }, 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG [...] 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:18:37 DEBUG }, 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 1713392317, 23:18:37 DEBUG "8.024910196171247" 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713392317" 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG --- status code --- 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 DEBUG --- http response --- 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG { 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG "data": { 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG "result": [ 23:18:37 DEBUG { 23:18:37 DEBUG "metric": { 23:18:37 DEBUG "pod": "ds-idrepo-0" 23:18:37 DEBUG }, 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG [...] 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG "pod": "ds-cts-2" 23:18:37 DEBUG }, 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 1713392317, 23:18:37 DEBUG "0" 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392317" 23:18:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392317" 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG --- status code --- 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 DEBUG --- http response --- 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG { 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG "data": { 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG "result": [ 23:18:37 DEBUG { 23:18:37 DEBUG "metric": { 23:18:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:18:37 DEBUG }, 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG [...] 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:18:37 DEBUG }, 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 1713392317, 23:18:37 DEBUG "196605" 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG --- status code --- 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 DEBUG --- http response --- 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG { 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG "data": { 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG "result": [ 23:18:37 DEBUG { 23:18:37 DEBUG "metric": { 23:18:37 DEBUG "instance": "10.142.0.107:9100" 23:18:37 DEBUG }, 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG [...] 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG "instance": "10.142.0.43:9100" 23:18:37 DEBUG }, 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 1713392317, 23:18:37 DEBUG "0.021000000000000234" 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392317" 23:18:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392317" 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG --- status code --- 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 DEBUG --- http response --- 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG { 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG "data": { 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG "result": [ 23:18:37 DEBUG { 23:18:37 DEBUG "metric": { 23:18:37 DEBUG "pod": "ds-idrepo-0" 23:18:37 DEBUG }, 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG [...] 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG "pod": "ds-idrepo-2" 23:18:37 DEBUG }, 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 1713392317, 23:18:37 DEBUG "0" 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392317" 23:18:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392317" 23:18:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713392317" 23:18:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392317" 23:18:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392317" 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG --- status code --- 23:18:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392317" 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG --- http response --- 23:18:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392317" 23:18:37 DEBUG --- status code --- 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG --- status code --- 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG --- http response --- 23:18:37 DEBUG --- status code --- 23:18:37 DEBUG --- status code --- 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 DEBUG { 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 DEBUG --- http response --- 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG "data": { 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG --- http response --- 23:18:37 DEBUG --- http response --- 23:18:37 DEBUG --- status code --- 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG { 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 DEBUG "result": [ 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG --- status code --- 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG --- http response --- 23:18:37 DEBUG { 23:18:37 DEBUG "data": { 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 DEBUG { 23:18:37 DEBUG { 23:18:37 DEBUG { 23:18:37 DEBUG "metric": { 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG --- http response --- 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG "instance": "10.142.0.107:9100" 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG "result": [ 23:18:37 DEBUG "data": { 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:37 DEBUG "data": { 23:18:37 DEBUG "data": { 23:18:37 DEBUG }, 23:18:37 DEBUG { 23:18:37 DEBUG { 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG "value": [ 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG "metric": { 23:18:37 DEBUG "result": [ 23:18:37 DEBUG { 23:18:37 DEBUG "result": [ 23:18:37 DEBUG "result": [ 23:18:37 DEBUG 23:18:37 DEBUG "data": { 23:18:37 DEBUG "pod": "ds-idrepo-0" 23:18:37 DEBUG { 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG { 23:18:37 DEBUG { 23:18:37 DEBUG "metric": { 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG }, 23:18:37 INFO [http_cmd]: http status code OK 23:18:37 DEBUG "metric": { 23:18:37 DEBUG "data": { 23:18:37 DEBUG "metric": { 23:18:37 DEBUG 23:18:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:18:37 DEBUG "result": [ 23:18:37 DEBUG "value": [ 23:18:37 DEBUG --- status code --- 23:18:37 DEBUG "pod": "ds-cts-0" 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:18:37 DEBUG 23:18:37 DEBUG }, 23:18:37 DEBUG { 23:18:37 DEBUG 23:18:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:37 DEBUG }, 23:18:37 DEBUG "result": [ 23:18:37 DEBUG }, 23:18:37 DEBUG [...] 23:18:37 DEBUG "value": [ 23:18:37 DEBUG "metric": { 23:18:37 DEBUG 23:18:37 DEBUG --- http response --- 23:18:37 DEBUG "value": [ 23:18:37 DEBUG { 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG "pod": "ds-cts-0" 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:18:37 DEBUG "metric": { 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG }, 23:18:37 DEBUG [...] 23:18:37 DEBUG 23:18:37 DEBUG ----- output ----- 23:18:37 DEBUG "pod": "ds-idrepo-0" 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG "instance": "10.142.0.43:9100" 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG { 23:18:37 DEBUG }, 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG }, 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG [...] 23:18:37 DEBUG "status": "success", 23:18:37 DEBUG "value": [ 23:18:37 DEBUG [...] 23:18:37 DEBUG [...] 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG "data": { 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 1713392317, 23:18:37 DEBUG 23:18:37 DEBUG "pod": "ds-idrepo-2" 23:18:37 DEBUG 23:18:37 DEBUG "resultType": "vector", 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG "27282903.266666666" 23:18:37 DEBUG [...] 23:18:37 DEBUG }, 23:18:37 DEBUG 23:18:37 DEBUG "result": [ 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG 23:18:37 DEBUG ] 23:18:37 DEBUG 23:18:37 DEBUG "value": [ 23:18:37 DEBUG "pod": "ds-cts-2" 23:18:37 DEBUG { 23:18:37 DEBUG [...] 23:18:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:18:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:18:37 DEBUG } 23:18:37 DEBUG 23:18:37 DEBUG 1713392317, 23:18:37 DEBUG }, 23:18:37 DEBUG "metric": { 23:18:37 DEBUG 23:18:37 DEBUG }, 23:18:37 DEBUG }, 23:18:37 DEBUG ] 23:18:37 DEBUG 23:18:37 DEBUG "pod": "ds-cts-2" 23:18:37 DEBUG "value": [ 23:18:37 DEBUG "pod": "ds-idrepo-1" 23:18:37 DEBUG 23:18:37 DEBUG "value": [ 23:18:37 DEBUG "value": [ 23:18:37 DEBUG } 23:18:37 DEBUG "0" 23:18:37 DEBUG }, 23:18:37 DEBUG 1713392317, 23:18:37 DEBUG }, 23:18:37 DEBUG 23:18:37 DEBUG 1713392317, 23:18:37 DEBUG 1713392317, 23:18:37 DEBUG "0" 23:18:37 DEBUG ] 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 1713392317, 23:18:37 DEBUG "value": [ 23:18:37 DEBUG "pod": "ds-idrepo-2" 23:18:37 DEBUG "0.02" 23:18:37 DEBUG } 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG "0" 23:18:37 DEBUG "0" 23:18:37 DEBUG 23:18:37 DEBUG }, 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG ] 23:18:37 DEBUG ] 23:18:37 DEBUG ] 23:18:37 DEBUG 23:18:37 DEBUG "value": [ 23:18:37 DEBUG } 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 DEBUG 23:18:37 DEBUG 1713392317, 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG [...] 23:18:37 DEBUG "0" 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 DEBUG ] 23:18:37 DEBUG 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 DEBUG 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 DEBUG 23:18:37 DEBUG ] 23:18:37 DEBUG "pod": "ds-idrepo-2" 23:18:37 DEBUG } 23:18:37 DEBUG }, 23:18:37 DEBUG } 23:18:37 DEBUG "value": [ 23:18:37 DEBUG 1713392317, 23:18:37 DEBUG "2993.010581130438" 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG ] 23:18:37 DEBUG } 23:18:37 DEBUG } 23:18:37 INFO 23:18:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:18:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:18:37 INFO [loop_until]: OK (rc = 0) 23:18:37 DEBUG --- stdout --- 23:18:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5442Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2842m 9% 6158Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 5986m 20% 11130Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 2996Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 3920m 13% 10303Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 4933Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 6160m 20% 9998Mi 8% gke-xlou-cdm-ds-0f601751-cf7b 15397m 51% 3693Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 5599m 18% 9607Mi 8% gke-xlou-cdm-ds-0f601751-j488 2864m 9% 5512Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 3382m 11% 11388Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 3031m 10% 5995Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 3694m 12% 9753Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 8202m 51% 11144Mi 19% 23:18:37 DEBUG --- stderr --- 23:18:37 DEBUG 23:19:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392374" 23:19:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713392374" 23:19:34 ERROR [http_cmd]: ERROR 23:19:34 INFO [http_cmd]: http status code OK 23:19:34 ERROR --- status code --- 23:19:34 DEBUG --- status code --- 23:19:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:19:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:34 ERROR --- http response --- 23:19:34 DEBUG --- http response --- 23:19:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:19:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:19:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:19:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392375" 23:19:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392375" 23:19:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392375" 23:19:35 INFO [http_cmd]: http status code OK 23:19:35 DEBUG --- status code --- 23:19:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:35 DEBUG --- http response --- 23:19:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:35 DEBUG ----- output ----- 23:19:35 DEBUG { 23:19:35 DEBUG "status": "success", 23:19:35 DEBUG "data": { 23:19:35 DEBUG "resultType": "vector", 23:19:35 DEBUG "result": [ 23:19:35 DEBUG { 23:19:35 DEBUG "metric": { 23:19:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG [...] 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 1713392375, 23:19:35 DEBUG "998.3" 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG } 23:19:35 INFO [http_cmd]: http status code OK 23:19:35 DEBUG --- status code --- 23:19:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:35 DEBUG --- http response --- 23:19:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:19:35 DEBUG ----- output ----- 23:19:35 DEBUG { 23:19:35 DEBUG "status": "success", 23:19:35 DEBUG "data": { 23:19:35 DEBUG "resultType": "vector", 23:19:35 DEBUG "result": [ 23:19:35 DEBUG { 23:19:35 DEBUG "metric": { 23:19:35 DEBUG "pod": "ds-idrepo-1" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG [...] 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 1713392375, 23:19:35 DEBUG "0.003083510604010027" 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG } 23:19:35 INFO [http_cmd]: http status code OK 23:19:35 DEBUG --- status code --- 23:19:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:35 DEBUG --- http response --- 23:19:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:35 DEBUG ----- output ----- 23:19:35 DEBUG { 23:19:35 DEBUG "status": "success", 23:19:35 DEBUG "data": { 23:19:35 DEBUG "resultType": "vector", 23:19:35 DEBUG "result": [ 23:19:35 DEBUG { 23:19:35 DEBUG "metric": { 23:19:35 DEBUG "instance": "10.142.0.107:9100" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG [...] 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG "instance": "10.142.0.43:9100" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 1713392375, 23:19:35 DEBUG "0.04256666666666434" 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG } 23:19:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392375" 23:19:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713392375" 23:19:35 INFO [http_cmd]: http status code OK 23:19:35 DEBUG --- status code --- 23:19:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:35 DEBUG --- http response --- 23:19:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:35 DEBUG ----- output ----- 23:19:35 DEBUG { 23:19:35 DEBUG "status": "success", 23:19:35 DEBUG "data": { 23:19:35 DEBUG "resultType": "vector", 23:19:35 DEBUG "result": [ 23:19:35 DEBUG { 23:19:35 DEBUG "metric": { 23:19:35 DEBUG "instance": "10.142.0.107:9100" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG [...] 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG "instance": "10.142.0.43:9100" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 1713392375, 23:19:35 DEBUG "36062193.36666667" 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG } 23:19:35 INFO [http_cmd]: http status code OK 23:19:35 DEBUG --- status code --- 23:19:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:35 DEBUG --- http response --- 23:19:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:35 DEBUG ----- output ----- 23:19:35 DEBUG { 23:19:35 DEBUG "status": "success", 23:19:35 DEBUG "data": { 23:19:35 DEBUG "resultType": "vector", 23:19:35 DEBUG "result": [ 23:19:35 DEBUG { 23:19:35 DEBUG "metric": { 23:19:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG [...] 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 1713392375, 23:19:35 DEBUG "8337444864" 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG } 23:19:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392375" 23:19:35 INFO [http_cmd]: http status code OK 23:19:35 DEBUG --- status code --- 23:19:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:35 DEBUG --- http response --- 23:19:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:19:35 DEBUG ----- output ----- 23:19:35 DEBUG { 23:19:35 DEBUG "status": "success", 23:19:35 DEBUG "data": { 23:19:35 DEBUG "resultType": "vector", 23:19:35 DEBUG "result": [ 23:19:35 DEBUG { 23:19:35 DEBUG "metric": { 23:19:35 DEBUG "pod": "ds-cts-0" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG [...] 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG "pod": "ds-cts-2" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 1713392375, 23:19:35 DEBUG "29.966666666666665" 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG } 23:19:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392375" 23:19:35 INFO [http_cmd]: http status code OK 23:19:35 DEBUG --- status code --- 23:19:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:35 DEBUG --- http response --- 23:19:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:35 DEBUG ----- output ----- 23:19:35 DEBUG { 23:19:35 DEBUG "status": "success", 23:19:35 DEBUG "data": { 23:19:35 DEBUG "resultType": "vector", 23:19:35 DEBUG "result": [ 23:19:35 DEBUG { 23:19:35 DEBUG "metric": { 23:19:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG [...] 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG 23:19:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:19:35 DEBUG }, 23:19:35 DEBUG "value": [ 23:19:35 DEBUG 1713392375, 23:19:35 DEBUG "1355.5" 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG ] 23:19:35 DEBUG } 23:19:35 DEBUG } 23:19:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392376" 23:19:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392376" 23:19:36 INFO [http_cmd]: http status code OK 23:19:36 INFO [http_cmd]: http status code OK 23:19:36 DEBUG --- status code --- 23:19:36 DEBUG --- status code --- 23:19:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:36 DEBUG --- http response --- 23:19:36 DEBUG --- http response --- 23:19:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:19:36 DEBUG ----- output ----- 23:19:36 DEBUG ----- output ----- 23:19:36 DEBUG { 23:19:36 DEBUG { 23:19:36 DEBUG "status": "success", 23:19:36 DEBUG "status": "success", 23:19:36 DEBUG "data": { 23:19:36 DEBUG "data": { 23:19:36 DEBUG "resultType": "vector", 23:19:36 DEBUG "resultType": "vector", 23:19:36 DEBUG "result": [ 23:19:36 DEBUG "result": [ 23:19:36 DEBUG { 23:19:36 DEBUG { 23:19:36 DEBUG "metric": { 23:19:36 DEBUG "metric": { 23:19:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:19:36 DEBUG "pod": "ds-idrepo-1" 23:19:36 DEBUG }, 23:19:36 DEBUG }, 23:19:36 DEBUG "value": [ 23:19:36 DEBUG "value": [ 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG [...] 23:19:36 DEBUG [...] 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:19:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:19:36 DEBUG }, 23:19:36 DEBUG }, 23:19:36 DEBUG "value": [ 23:19:36 DEBUG "value": [ 23:19:36 DEBUG 1713392376, 23:19:36 DEBUG 1713392376, 23:19:36 DEBUG "998.3" 23:19:36 DEBUG "34393409.39856894" 23:19:36 DEBUG ] 23:19:36 DEBUG ] 23:19:36 DEBUG } 23:19:36 DEBUG } 23:19:36 DEBUG ] 23:19:36 DEBUG ] 23:19:36 DEBUG } 23:19:36 DEBUG } 23:19:36 DEBUG } 23:19:36 DEBUG } 23:19:36 INFO 23:19:36 INFO [loop_until]: kubectl --namespace=xlou top pods 23:19:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:19:36 INFO [loop_until]: OK (rc = 0) 23:19:36 DEBUG --- stdout --- 23:19:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 7161m 7395Mi am-b59c765b7-bgsxp 6148m 7757Mi am-b59c765b7-ch5q4 6701m 8541Mi ds-cts-0 6818m 8009Mi ds-cts-1 6235m 7817Mi ds-cts-2 7347m 8037Mi ds-idrepo-0 2949m 3222Mi ds-idrepo-1 2942m 2880Mi ds-idrepo-2 2937m 2935Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 9m 971Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7806m 7827Mi 23:19:36 DEBUG --- stderr --- 23:19:36 DEBUG 23:19:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392376" 23:19:36 INFO [http_cmd]: http status code OK 23:19:36 DEBUG --- status code --- 23:19:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:36 DEBUG --- http response --- 23:19:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:36 DEBUG ----- output ----- 23:19:36 DEBUG { 23:19:36 DEBUG "status": "success", 23:19:36 DEBUG "data": { 23:19:36 DEBUG "resultType": "vector", 23:19:36 DEBUG "result": [ 23:19:36 DEBUG { 23:19:36 DEBUG "metric": { 23:19:36 DEBUG "instance": "10.142.0.107:9100" 23:19:36 DEBUG }, 23:19:36 DEBUG "value": [ 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG [...] 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG 23:19:36 DEBUG "instance": "10.142.0.43:9100" 23:19:36 DEBUG }, 23:19:36 DEBUG "value": [ 23:19:36 DEBUG 1713392376, 23:19:36 DEBUG "252563092480" 23:19:36 DEBUG ] 23:19:36 DEBUG } 23:19:36 DEBUG ] 23:19:36 DEBUG } 23:19:36 DEBUG } 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392377" 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG --- http response --- 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG { 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG "data": { 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG "result": [ 23:19:37 DEBUG { 23:19:37 DEBUG "metric": { 23:19:37 DEBUG "pod": "ds-idrepo-1" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG [...] 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "0" 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG } 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392377" 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG --- http response --- 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG { 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG "data": { 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG "result": [ 23:19:37 DEBUG { 23:19:37 DEBUG "metric": { 23:19:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG [...] 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "998.1" 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG } 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713392377" 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG --- http response --- 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713392377" 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG { 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG "data": { 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG "result": [ 23:19:37 DEBUG { 23:19:37 DEBUG "metric": { 23:19:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG [...] 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "8.041623028864421" 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG } 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG --- http response --- 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG { 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG "data": { 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG "result": [ 23:19:37 DEBUG { 23:19:37 DEBUG "metric": { 23:19:37 DEBUG "pod": "ds-idrepo-0" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG [...] 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG "pod": "ds-cts-2" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "0" 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG } 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392377" 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392377" 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG --- http response --- 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG { 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG "data": { 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG "result": [ 23:19:37 DEBUG { 23:19:37 DEBUG "metric": { 23:19:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG [...] 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "191815.5" 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG } 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392377" 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392377" 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG --- http response --- 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG { 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG "data": { 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG "result": [ 23:19:37 DEBUG { 23:19:37 DEBUG "metric": { 23:19:37 DEBUG "instance": "10.142.0.107:9100" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG [...] 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG "instance": "10.142.0.43:9100" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "0.03266666666666739" 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG } 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392377" 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG --- http response --- 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG { 23:19:37 DEBUG --- http response --- 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG "data": { 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG "result": [ 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:37 DEBUG { 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG "metric": { 23:19:37 DEBUG { 23:19:37 DEBUG "pod": "ds-idrepo-0" 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG }, 23:19:37 DEBUG "data": { 23:19:37 DEBUG "value": [ 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG 23:19:37 DEBUG "result": [ 23:19:37 DEBUG 23:19:37 DEBUG { 23:19:37 DEBUG 23:19:37 DEBUG "metric": { 23:19:37 DEBUG [...] 23:19:37 DEBUG "instance": "10.142.0.107:9100" 23:19:37 DEBUG 23:19:37 DEBUG }, 23:19:37 DEBUG 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG "pod": "ds-idrepo-2" 23:19:37 DEBUG 23:19:37 DEBUG }, 23:19:37 DEBUG 23:19:37 DEBUG "value": [ 23:19:37 DEBUG [...] 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG 23:19:37 DEBUG "0" 23:19:37 DEBUG 23:19:37 DEBUG ] 23:19:37 DEBUG 23:19:37 DEBUG } 23:19:37 DEBUG "instance": "10.142.0.43:9100" 23:19:37 DEBUG ] 23:19:37 DEBUG }, 23:19:37 DEBUG } 23:19:37 DEBUG "value": [ 23:19:37 DEBUG } 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "36062220.49999999" 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG } 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG --- http response --- 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG { 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG "data": { 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG "result": [ 23:19:37 DEBUG { 23:19:37 DEBUG "metric": { 23:19:37 DEBUG "pod": "ds-idrepo-0" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG [...] 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG "pod": "ds-idrepo-2" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "0" 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG } 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392377" 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG --- http response --- 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG { 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG "data": { 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG "result": [ 23:19:37 DEBUG { 23:19:37 DEBUG "metric": { 23:19:37 DEBUG "pod": "ds-cts-0" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG [...] 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG "pod": "ds-cts-2" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "0.00019099333333333335" 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG } 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392377" 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG --- http response --- 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG { 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG "data": { 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG "result": [ 23:19:37 DEBUG { 23:19:37 DEBUG "metric": { 23:19:37 DEBUG "pod": "ds-idrepo-1" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG [...] 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "528.0732289047895" 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG } 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392377" 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392377" 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713392377" 23:19:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392377" 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG --- http response --- 23:19:37 DEBUG --- http response --- 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG { 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG "data": { 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG "result": [ 23:19:37 DEBUG { 23:19:37 DEBUG "metric": { 23:19:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:19:37 DEBUG [...] 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG 23:19:37 DEBUG { 23:19:37 DEBUG 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG 23:19:37 DEBUG "data": { 23:19:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG }, 23:19:37 DEBUG "result": [ 23:19:37 DEBUG "value": [ 23:19:37 DEBUG { 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "metric": { 23:19:37 DEBUG "0" 23:19:37 DEBUG "pod": "ds-cts-0" 23:19:37 DEBUG ] 23:19:37 DEBUG }, 23:19:37 DEBUG } 23:19:37 DEBUG "value": [ 23:19:37 DEBUG ] 23:19:37 DEBUG 23:19:37 DEBUG } 23:19:37 DEBUG 23:19:37 DEBUG } 23:19:37 DEBUG 23:19:37 DEBUG [...] 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG "pod": "ds-cts-2" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "0" 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG } 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG --- http response --- 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG { 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG "data": { 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG "result": [ 23:19:37 DEBUG { 23:19:37 DEBUG "metric": { 23:19:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG [...] 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "0.02" 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG } 23:19:37 INFO [http_cmd]: http status code OK 23:19:37 DEBUG --- status code --- 23:19:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:37 DEBUG --- http response --- 23:19:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:37 DEBUG ----- output ----- 23:19:37 DEBUG { 23:19:37 DEBUG "status": "success", 23:19:37 DEBUG "data": { 23:19:37 DEBUG "resultType": "vector", 23:19:37 DEBUG "result": [ 23:19:37 DEBUG { 23:19:37 DEBUG "metric": { 23:19:37 DEBUG "pod": "ds-idrepo-0" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG [...] 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG 23:19:37 DEBUG "pod": "ds-idrepo-2" 23:19:37 DEBUG }, 23:19:37 DEBUG "value": [ 23:19:37 DEBUG 1713392377, 23:19:37 DEBUG "0" 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG ] 23:19:37 DEBUG } 23:19:37 DEBUG } 23:19:37 INFO 23:19:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:19:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:19:37 INFO [loop_until]: OK (rc = 0) 23:19:37 DEBUG --- stdout --- 23:19:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5437Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3070m 10% 6143Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6485m 21% 11139Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 2995Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 6784m 22% 10292Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 4956Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 6822m 22% 10009Mi 8% gke-xlou-cdm-ds-0f601751-cf7b 14826m 49% 3709Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 6612m 22% 9619Mi 8% gke-xlou-cdm-ds-0f601751-j488 3185m 10% 5506Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 6652m 22% 11395Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 3185m 10% 5992Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 7220m 24% 9753Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 8327m 52% 10982Mi 18% 23:19:37 DEBUG --- stderr --- 23:19:37 DEBUG 23:20:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713392434" 23:20:34 ERROR [http_cmd]: ERROR 23:20:34 ERROR --- status code --- 23:20:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:20:34 ERROR --- http response --- 23:20:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:20:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:20:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392434" 23:20:34 INFO [http_cmd]: http status code OK 23:20:34 DEBUG --- status code --- 23:20:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:34 DEBUG --- http response --- 23:20:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:20:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392435" 23:20:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392435" 23:20:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392435" 23:20:35 INFO [http_cmd]: http status code OK 23:20:35 DEBUG --- status code --- 23:20:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:35 DEBUG --- http response --- 23:20:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:35 DEBUG ----- output ----- 23:20:35 DEBUG { 23:20:35 DEBUG "status": "success", 23:20:35 DEBUG "data": { 23:20:35 DEBUG "resultType": "vector", 23:20:35 DEBUG "result": [ 23:20:35 DEBUG { 23:20:35 DEBUG "metric": { 23:20:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG [...] 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 1713392435, 23:20:35 DEBUG "897.1333333333333" 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG } 23:20:35 INFO [http_cmd]: http status code OK 23:20:35 DEBUG --- status code --- 23:20:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:35 DEBUG --- http response --- 23:20:35 INFO [http_cmd]: http status code OK 23:20:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:35 DEBUG --- status code --- 23:20:35 DEBUG ----- output ----- 23:20:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:35 DEBUG { 23:20:35 DEBUG --- http response --- 23:20:35 DEBUG "status": "success", 23:20:35 DEBUG "data": { 23:20:35 DEBUG "resultType": "vector", 23:20:35 DEBUG "result": [ 23:20:35 DEBUG { 23:20:35 DEBUG "metric": { 23:20:35 DEBUG "instance": "10.142.0.107:9100" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG [...] 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG "instance": "10.142.0.43:9100" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 1713392435, 23:20:35 DEBUG "0.029466666666665253" 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG } 23:20:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:35 DEBUG ----- output ----- 23:20:35 DEBUG { 23:20:35 DEBUG "status": "success", 23:20:35 DEBUG "data": { 23:20:35 DEBUG "resultType": "vector", 23:20:35 DEBUG "result": [ 23:20:35 DEBUG { 23:20:35 DEBUG "metric": { 23:20:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG [...] 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG "pod": "am-b59c765b7-7xg4f" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 1713392435, 23:20:35 DEBUG "0" 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG } 23:20:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392435" 23:20:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713392435" 23:20:35 INFO [http_cmd]: http status code OK 23:20:35 DEBUG --- status code --- 23:20:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:35 DEBUG --- http response --- 23:20:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:35 DEBUG ----- output ----- 23:20:35 DEBUG { 23:20:35 DEBUG "status": "success", 23:20:35 DEBUG "data": { 23:20:35 DEBUG "resultType": "vector", 23:20:35 DEBUG "result": [ 23:20:35 DEBUG { 23:20:35 DEBUG "metric": { 23:20:35 DEBUG "instance": "10.142.0.107:9100" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG [...] 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG "instance": "10.142.0.43:9100" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 1713392435, 23:20:35 DEBUG "33058605.4" 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG } 23:20:35 INFO [http_cmd]: http status code OK 23:20:35 DEBUG --- status code --- 23:20:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:35 DEBUG --- http response --- 23:20:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:35 DEBUG ----- output ----- 23:20:35 DEBUG { 23:20:35 DEBUG "status": "success", 23:20:35 DEBUG "data": { 23:20:35 DEBUG "resultType": "vector", 23:20:35 DEBUG "result": [ 23:20:35 DEBUG { 23:20:35 DEBUG "metric": { 23:20:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG [...] 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 1713392435, 23:20:35 DEBUG "8560197632" 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG } 23:20:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392435" 23:20:35 INFO [http_cmd]: http status code OK 23:20:35 DEBUG --- status code --- 23:20:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:35 DEBUG --- http response --- 23:20:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:20:35 DEBUG ----- output ----- 23:20:35 DEBUG { 23:20:35 DEBUG "status": "success", 23:20:35 DEBUG "data": { 23:20:35 DEBUG "resultType": "vector", 23:20:35 DEBUG "result": [ 23:20:35 DEBUG { 23:20:35 DEBUG "metric": { 23:20:35 DEBUG "pod": "ds-cts-0" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG [...] 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG "pod": "ds-cts-2" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 1713392435, 23:20:35 DEBUG "29.966666666666665" 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG } 23:20:35 WARNING File /tmp/lodemon_data-20240417_224222/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [17/Apr/2024 23:20:35] "GET /monitoring/average?start_time=20240417_221645&stop_time=20240417_221834 HTTP/1.1" 200 - 23:20:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392435" 23:20:35 INFO [http_cmd]: http status code OK 23:20:35 DEBUG --- status code --- 23:20:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:35 DEBUG --- http response --- 23:20:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:35 DEBUG ----- output ----- 23:20:35 DEBUG { 23:20:35 DEBUG "status": "success", 23:20:35 DEBUG "data": { 23:20:35 DEBUG "resultType": "vector", 23:20:35 DEBUG "result": [ 23:20:35 DEBUG { 23:20:35 DEBUG "metric": { 23:20:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG [...] 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG 23:20:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:20:35 DEBUG }, 23:20:35 DEBUG "value": [ 23:20:35 DEBUG 1713392435, 23:20:35 DEBUG "1090.7666666666667" 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG ] 23:20:35 DEBUG } 23:20:35 DEBUG } 23:20:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392436" 23:20:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392436" 23:20:36 INFO [http_cmd]: http status code OK 23:20:36 INFO [http_cmd]: http status code OK 23:20:36 DEBUG --- status code --- 23:20:36 DEBUG --- status code --- 23:20:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:36 DEBUG --- http response --- 23:20:36 DEBUG --- http response --- 23:20:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:36 DEBUG ----- output ----- 23:20:36 DEBUG { 23:20:36 DEBUG "status": "success", 23:20:36 DEBUG "data": { 23:20:36 DEBUG "resultType": "vector", 23:20:36 DEBUG "result": [ 23:20:36 DEBUG { 23:20:36 DEBUG "metric": { 23:20:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:20:36 DEBUG }, 23:20:36 DEBUG "value": [ 23:20:36 DEBUG 23:20:36 DEBUG 23:20:36 DEBUG 23:20:36 DEBUG [...] 23:20:36 DEBUG 23:20:36 DEBUG 23:20:36 DEBUG 23:20:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:20:36 DEBUG }, 23:20:36 DEBUG "value": [ 23:20:36 DEBUG 1713392436, 23:20:36 DEBUG "897.1333333333333" 23:20:36 DEBUG ] 23:20:36 DEBUG } 23:20:36 DEBUG ] 23:20:36 DEBUG } 23:20:36 DEBUG } 23:20:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:20:36 DEBUG ----- output ----- 23:20:36 DEBUG { 23:20:36 DEBUG "status": "success", 23:20:36 DEBUG "data": { 23:20:36 DEBUG "resultType": "vector", 23:20:36 DEBUG "result": [ 23:20:36 DEBUG { 23:20:36 DEBUG "metric": { 23:20:36 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:20:36 DEBUG }, 23:20:36 DEBUG "value": [ 23:20:36 DEBUG 23:20:36 DEBUG 23:20:36 DEBUG 23:20:36 DEBUG [...] 23:20:36 DEBUG 23:20:36 DEBUG 23:20:36 DEBUG 23:20:36 DEBUG "pod": "am-b59c765b7-7xg4f" 23:20:36 DEBUG }, 23:20:36 DEBUG "value": [ 23:20:36 DEBUG 1713392436, 23:20:36 DEBUG "0" 23:20:36 DEBUG ] 23:20:36 DEBUG } 23:20:36 DEBUG ] 23:20:36 DEBUG } 23:20:36 DEBUG } 23:20:36 INFO 23:20:36 INFO [loop_until]: kubectl --namespace=xlou top pods 23:20:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:20:36 INFO [loop_until]: OK (rc = 0) 23:20:36 DEBUG --- stdout --- 23:20:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 4485m 8900Mi am-b59c765b7-bgsxp 4769m 7751Mi am-b59c765b7-ch5q4 5179m 8542Mi ds-cts-0 4880m 8012Mi ds-cts-1 5842m 7804Mi ds-cts-2 5448m 8039Mi ds-idrepo-0 2067m 3222Mi ds-idrepo-1 2760m 2880Mi ds-idrepo-2 2092m 2935Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 971Mi idm-b85f9d978-wnhxb 3m 1012Mi lodemon-7cfb6bd874-prp2f 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 5300m 15Mi 23:20:36 DEBUG --- stderr --- 23:20:36 DEBUG 23:20:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392436" 23:20:36 INFO [http_cmd]: http status code OK 23:20:36 DEBUG --- status code --- 23:20:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:36 DEBUG --- http response --- 23:20:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:36 DEBUG ----- output ----- 23:20:36 DEBUG { 23:20:36 DEBUG "status": "success", 23:20:36 DEBUG "data": { 23:20:36 DEBUG "resultType": "vector", 23:20:36 DEBUG "result": [ 23:20:36 DEBUG { 23:20:36 DEBUG "metric": { 23:20:36 DEBUG "instance": "10.142.0.107:9100" 23:20:36 DEBUG }, 23:20:36 DEBUG "value": [ 23:20:36 DEBUG 23:20:36 DEBUG 23:20:36 DEBUG 23:20:36 DEBUG [...] 23:20:36 DEBUG 23:20:36 DEBUG 23:20:36 DEBUG 23:20:36 DEBUG "instance": "10.142.0.43:9100" 23:20:36 DEBUG }, 23:20:36 DEBUG "value": [ 23:20:36 DEBUG 1713392436, 23:20:36 DEBUG "254423944192" 23:20:36 DEBUG ] 23:20:36 DEBUG } 23:20:36 DEBUG ] 23:20:36 DEBUG } 23:20:36 DEBUG } 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392437" 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "pod": "ds-idrepo-0" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "pod": "ds-idrepo-1" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "0" 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392437" 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "897.1333333333333" 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713392437" 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713392437" 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "pod": "ds-idrepo-0" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "pod": "ds-cts-2" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "0" 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "8.03843065737728" 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392437" 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "179701.5" 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392437" 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392437" 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392437" 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392437" 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "instance": "10.142.0.107:9100" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "instance": "10.142.0.43:9100" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "0.02066666666666655" 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "instance": "10.142.0.107:9100" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "instance": "10.142.0.43:9100" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "33057016.43333333" 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "pod": "ds-idrepo-0" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "pod": "ds-idrepo-2" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "0" 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "pod": "ds-idrepo-0" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "pod": "ds-idrepo-2" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "0" 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392437" 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "pod": "ds-cts-0" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "pod": "ds-cts-2" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "0.0006658266666666666" 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713392437" 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392437" 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392437" 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392437" 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "0.021" 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "0" 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392437" 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "pod": "ds-cts-0" 23:20:37 DEBUG "pod": "ds-idrepo-0" 23:20:37 DEBUG }, 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "pod": "ds-cts-2" 23:20:37 DEBUG "pod": "ds-idrepo-2" 23:20:37 DEBUG }, 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "0" 23:20:37 DEBUG "0" 23:20:37 DEBUG ] 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 INFO [http_cmd]: http status code OK 23:20:37 DEBUG --- status code --- 23:20:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:37 DEBUG --- http response --- 23:20:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:20:37 DEBUG ----- output ----- 23:20:37 DEBUG { 23:20:37 DEBUG "status": "success", 23:20:37 DEBUG "data": { 23:20:37 DEBUG "resultType": "vector", 23:20:37 DEBUG "result": [ 23:20:37 DEBUG { 23:20:37 DEBUG "metric": { 23:20:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG [...] 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG 23:20:37 DEBUG "pod": "ds-idrepo-1" 23:20:37 DEBUG }, 23:20:37 DEBUG "value": [ 23:20:37 DEBUG 1713392437, 23:20:37 DEBUG "1591.242662257131" 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG ] 23:20:37 DEBUG } 23:20:37 DEBUG } 23:20:37 INFO 23:20:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:20:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:20:37 INFO [loop_until]: OK (rc = 0) 23:20:37 DEBUG --- stdout --- 23:20:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5434Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2142m 7% 6148Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 3845m 12% 11138Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 2993Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 5041m 16% 10288Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 4952Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 3728m 12% 11527Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 10121m 33% 2988Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 4557m 15% 9600Mi 8% gke-xlou-cdm-ds-0f601751-j488 2424m 8% 5514Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 5396m 18% 11384Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 1875m 6% 5994Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 5392m 18% 9756Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 5783m 36% 3923Mi 6% 23:20:37 DEBUG --- stderr --- 23:20:37 DEBUG 23:21:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713392494" 23:21:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392494" 23:21:34 ERROR [http_cmd]: ERROR 23:21:34 ERROR --- status code --- 23:21:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:21:34 ERROR --- http response --- 23:21:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:21:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:21:34 INFO [http_cmd]: http status code OK 23:21:34 DEBUG --- status code --- 23:21:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:34 DEBUG --- http response --- 23:21:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:21:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392495" 23:21:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392495" 23:21:35 INFO [http_cmd]: http status code OK 23:21:35 DEBUG --- status code --- 23:21:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:35 DEBUG --- http response --- 23:21:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:21:35 DEBUG ----- output ----- 23:21:35 DEBUG { 23:21:35 DEBUG "status": "success", 23:21:35 DEBUG "data": { 23:21:35 DEBUG "resultType": "vector", 23:21:35 DEBUG "result": [ 23:21:35 DEBUG { 23:21:35 DEBUG "metric": { 23:21:35 DEBUG "pod": "ds-idrepo-0" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG [...] 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 1713392495, 23:21:35 DEBUG "0" 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG } 23:21:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392495" 23:21:35 INFO [http_cmd]: http status code OK 23:21:35 DEBUG --- status code --- 23:21:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:35 DEBUG --- http response --- 23:21:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:35 DEBUG ----- output ----- 23:21:35 DEBUG { 23:21:35 DEBUG "status": "success", 23:21:35 DEBUG "data": { 23:21:35 DEBUG "resultType": "vector", 23:21:35 DEBUG "result": [ 23:21:35 DEBUG { 23:21:35 DEBUG "metric": { 23:21:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG [...] 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 1713392495, 23:21:35 DEBUG "0.06666666666666667" 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG } 23:21:35 INFO [http_cmd]: http status code OK 23:21:35 DEBUG --- status code --- 23:21:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:35 DEBUG --- http response --- 23:21:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:35 DEBUG ----- output ----- 23:21:35 DEBUG { 23:21:35 DEBUG "status": "success", 23:21:35 DEBUG "data": { 23:21:35 DEBUG "resultType": "vector", 23:21:35 DEBUG "result": [ 23:21:35 DEBUG { 23:21:35 DEBUG "metric": { 23:21:35 DEBUG "instance": "10.142.0.107:9100" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG [...] 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG "instance": "10.142.0.43:9100" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 1713392495, 23:21:35 DEBUG "0.014700000000000085" 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG } 23:21:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392495" 23:21:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713392495" 23:21:35 INFO [http_cmd]: http status code OK 23:21:35 DEBUG --- status code --- 23:21:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:35 DEBUG --- http response --- 23:21:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:35 DEBUG ----- output ----- 23:21:35 DEBUG { 23:21:35 DEBUG "status": "success", 23:21:35 DEBUG "data": { 23:21:35 DEBUG "resultType": "vector", 23:21:35 DEBUG "result": [ 23:21:35 DEBUG { 23:21:35 DEBUG "metric": { 23:21:35 DEBUG "instance": "10.142.0.107:9100" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG [...] 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG "instance": "10.142.0.43:9100" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 1713392495, 23:21:35 DEBUG "7552145.7" 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG } 23:21:35 INFO [http_cmd]: http status code OK 23:21:35 DEBUG --- status code --- 23:21:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:35 DEBUG --- http response --- 23:21:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:35 DEBUG ----- output ----- 23:21:35 DEBUG { 23:21:35 DEBUG "status": "success", 23:21:35 DEBUG "data": { 23:21:35 DEBUG "resultType": "vector", 23:21:35 DEBUG "result": [ 23:21:35 DEBUG { 23:21:35 DEBUG "metric": { 23:21:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG [...] 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 1713392495, 23:21:35 DEBUG "111689728" 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG } 23:21:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392495" 23:21:35 INFO [http_cmd]: http status code OK 23:21:35 DEBUG --- status code --- 23:21:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:35 DEBUG --- http response --- 23:21:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:21:35 DEBUG ----- output ----- 23:21:35 DEBUG { 23:21:35 DEBUG "status": "success", 23:21:35 DEBUG "data": { 23:21:35 DEBUG "resultType": "vector", 23:21:35 DEBUG "result": [ 23:21:35 DEBUG { 23:21:35 DEBUG "metric": { 23:21:35 DEBUG "pod": "ds-cts-0" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG [...] 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG "pod": "ds-cts-2" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 1713392495, 23:21:35 DEBUG "30" 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG } 23:21:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392495" 23:21:35 INFO [http_cmd]: http status code OK 23:21:35 DEBUG --- status code --- 23:21:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:35 DEBUG --- http response --- 23:21:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:35 DEBUG ----- output ----- 23:21:35 DEBUG { 23:21:35 DEBUG "status": "success", 23:21:35 DEBUG "data": { 23:21:35 DEBUG "resultType": "vector", 23:21:35 DEBUG "result": [ 23:21:35 DEBUG { 23:21:35 DEBUG "metric": { 23:21:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG [...] 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG 23:21:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:21:35 DEBUG }, 23:21:35 DEBUG "value": [ 23:21:35 DEBUG 1713392495, 23:21:35 DEBUG "1047.9666666666667" 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG ] 23:21:35 DEBUG } 23:21:35 DEBUG } 23:21:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392496" 23:21:36 INFO [http_cmd]: http status code OK 23:21:36 DEBUG --- status code --- 23:21:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:36 DEBUG --- http response --- 23:21:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:21:36 DEBUG ----- output ----- 23:21:36 DEBUG { 23:21:36 DEBUG "status": "success", 23:21:36 DEBUG "data": { 23:21:36 DEBUG "resultType": "vector", 23:21:36 DEBUG "result": [ 23:21:36 DEBUG { 23:21:36 DEBUG "metric": { 23:21:36 DEBUG "pod": "ds-idrepo-0" 23:21:36 DEBUG }, 23:21:36 DEBUG "value": [ 23:21:36 DEBUG 23:21:36 DEBUG 23:21:36 DEBUG 23:21:36 DEBUG [...] 23:21:36 DEBUG 23:21:36 DEBUG 23:21:36 DEBUG 23:21:36 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:21:36 DEBUG }, 23:21:36 DEBUG "value": [ 23:21:36 DEBUG 1713392496, 23:21:36 DEBUG "0" 23:21:36 DEBUG ] 23:21:36 DEBUG } 23:21:36 DEBUG ] 23:21:36 DEBUG } 23:21:36 DEBUG } 23:21:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392496" 23:21:36 INFO [http_cmd]: http status code OK 23:21:36 DEBUG --- status code --- 23:21:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:36 DEBUG --- http response --- 23:21:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:36 DEBUG ----- output ----- 23:21:36 DEBUG { 23:21:36 DEBUG "status": "success", 23:21:36 DEBUG "data": { 23:21:36 DEBUG "resultType": "vector", 23:21:36 DEBUG "result": [ 23:21:36 DEBUG { 23:21:36 DEBUG "metric": { 23:21:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:21:36 DEBUG }, 23:21:36 DEBUG "value": [ 23:21:36 DEBUG 23:21:36 DEBUG 23:21:36 DEBUG 23:21:36 DEBUG [...] 23:21:36 DEBUG 23:21:36 DEBUG 23:21:36 DEBUG 23:21:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:21:36 DEBUG }, 23:21:36 DEBUG "value": [ 23:21:36 DEBUG 1713392496, 23:21:36 DEBUG "0.06666666666666667" 23:21:36 DEBUG ] 23:21:36 DEBUG } 23:21:36 DEBUG ] 23:21:36 DEBUG } 23:21:36 DEBUG } 23:21:36 INFO 23:21:36 INFO [loop_until]: kubectl --namespace=xlou top pods 23:21:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:21:36 INFO [loop_until]: OK (rc = 0) 23:21:36 DEBUG --- stdout --- 23:21:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 1177m 8882Mi am-b59c765b7-bgsxp 1154m 7743Mi am-b59c765b7-ch5q4 1205m 8541Mi ds-cts-0 3113m 8026Mi ds-cts-1 2796m 7794Mi ds-cts-2 3410m 8034Mi ds-idrepo-0 799m 3240Mi ds-idrepo-1 725m 2884Mi ds-idrepo-2 632m 2946Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 6m 972Mi idm-b85f9d978-wnhxb 4m 1012Mi lodemon-7cfb6bd874-prp2f 2m 64Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 802m 273Mi 23:21:36 DEBUG --- stderr --- 23:21:36 DEBUG 23:21:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392496" 23:21:36 INFO [http_cmd]: http status code OK 23:21:36 DEBUG --- status code --- 23:21:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:36 DEBUG --- http response --- 23:21:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:36 DEBUG ----- output ----- 23:21:36 DEBUG { 23:21:36 DEBUG "status": "success", 23:21:36 DEBUG "data": { 23:21:36 DEBUG "resultType": "vector", 23:21:36 DEBUG "result": [ 23:21:36 DEBUG { 23:21:36 DEBUG "metric": { 23:21:36 DEBUG "instance": "10.142.0.107:9100" 23:21:36 DEBUG }, 23:21:36 DEBUG "value": [ 23:21:36 DEBUG 23:21:36 DEBUG 23:21:36 DEBUG 23:21:36 DEBUG [...] 23:21:36 DEBUG 23:21:36 DEBUG 23:21:36 DEBUG 23:21:36 DEBUG "instance": "10.142.0.43:9100" 23:21:36 DEBUG }, 23:21:36 DEBUG "value": [ 23:21:36 DEBUG 1713392496, 23:21:36 DEBUG "255944145920" 23:21:36 DEBUG ] 23:21:36 DEBUG } 23:21:36 DEBUG ] 23:21:36 DEBUG } 23:21:36 DEBUG } 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392497" 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG { 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG "data": { 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG "result": [ 23:21:37 DEBUG { 23:21:37 DEBUG "metric": { 23:21:37 DEBUG "pod": "ds-idrepo-0" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG [...] 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG "pod": "ds-idrepo-2" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG "0" 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392497" 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG { 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG "data": { 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG "result": [ 23:21:37 DEBUG { 23:21:37 DEBUG "metric": { 23:21:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG [...] 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG "0.13333333333333333" 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713392497" 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713392497" 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG { 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG "data": { 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG "result": [ 23:21:37 DEBUG { 23:21:37 DEBUG "metric": { 23:21:37 DEBUG "pod": "ds-idrepo-0" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG [...] 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG "pod": "ds-cts-2" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG "0" 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG { 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG "data": { 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG "result": [ 23:21:37 DEBUG { 23:21:37 DEBUG "metric": { 23:21:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG [...] 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG "0.03302844079368143" 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392497" 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG { 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG "data": { 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG "result": [ 23:21:37 DEBUG { 23:21:37 DEBUG "metric": { 23:21:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG [...] 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG "130494.5" 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392497" 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG { 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG "data": { 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG "result": [ 23:21:37 DEBUG { 23:21:37 DEBUG "metric": { 23:21:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG [...] 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG "0" 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392497" 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG { 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG "data": { 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG "result": [ 23:21:37 DEBUG { 23:21:37 DEBUG "metric": { 23:21:37 DEBUG "pod": "ds-cts-0" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG [...] 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG "pod": "ds-cts-2" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG "0" 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392497" 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392497" 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392497" 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392497" 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392497" 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392497" 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713392497" 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG { 23:21:37 DEBUG { 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG "data": { 23:21:37 DEBUG "data": { 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG "result": [ 23:21:37 DEBUG "result": [ 23:21:37 DEBUG { 23:21:37 DEBUG { 23:21:37 DEBUG "metric": { 23:21:37 DEBUG "metric": { 23:21:37 DEBUG "pod": "ds-idrepo-0" 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 DEBUG "pod": "ds-idrepo-0" 23:21:37 DEBUG }, 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:37 DEBUG 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 DEBUG 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG [...] 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG [...] 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG { 23:21:37 DEBUG 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG "data": { 23:21:37 DEBUG 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:21:37 DEBUG 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG "pod": "ds-idrepo-2" 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG "pod": "ds-idrepo-2" 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG "result": [ 23:21:37 DEBUG }, 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG { 23:21:37 DEBUG }, 23:21:37 DEBUG { 23:21:37 DEBUG { 23:21:37 DEBUG { 23:21:37 DEBUG "value": [ 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:37 DEBUG "value": [ 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG "metric": { 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG "data": { 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG "data": { 23:21:37 DEBUG "data": { 23:21:37 DEBUG "instance": "10.142.0.107:9100" 23:21:37 DEBUG "0.06666666666666667" 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG { 23:21:37 DEBUG "0.06666666666666667" 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG }, 23:21:37 DEBUG ] 23:21:37 DEBUG "result": [ 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG ] 23:21:37 DEBUG "result": [ 23:21:37 DEBUG "result": [ 23:21:37 DEBUG "value": [ 23:21:37 DEBUG } 23:21:37 DEBUG { 23:21:37 DEBUG "data": { 23:21:37 DEBUG } 23:21:37 DEBUG { 23:21:37 DEBUG { 23:21:37 DEBUG 23:21:37 DEBUG ] 23:21:37 DEBUG "metric": { 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG ] 23:21:37 DEBUG "metric": { 23:21:37 DEBUG "metric": { 23:21:37 DEBUG 23:21:37 DEBUG } 23:21:37 DEBUG "pod": "ds-cts-0" 23:21:37 DEBUG "result": [ 23:21:37 DEBUG } 23:21:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:21:37 DEBUG "instance": "10.142.0.107:9100" 23:21:37 DEBUG 23:21:37 DEBUG } 23:21:37 DEBUG }, 23:21:37 DEBUG { 23:21:37 DEBUG } 23:21:37 DEBUG }, 23:21:37 DEBUG }, 23:21:37 DEBUG [...] 23:21:37 DEBUG "value": [ 23:21:37 DEBUG "metric": { 23:21:37 DEBUG "value": [ 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG }, 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG [...] 23:21:37 DEBUG [...] 23:21:37 DEBUG 23:21:37 DEBUG [...] 23:21:37 DEBUG "instance": "10.142.0.43:9100" 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG }, 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG [...] 23:21:37 DEBUG 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG "instance": "10.142.0.43:9100" 23:21:37 DEBUG "pod": "ds-cts-2" 23:21:37 DEBUG 23:21:37 DEBUG "pod": "ds-idrepo-2" 23:21:37 DEBUG "0.009666666666666579" 23:21:37 DEBUG }, 23:21:37 DEBUG }, 23:21:37 DEBUG 23:21:37 DEBUG }, 23:21:37 DEBUG ] 23:21:37 DEBUG "value": [ 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 23:21:37 DEBUG "value": [ 23:21:37 DEBUG } 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG ] 23:21:37 DEBUG "7533907.533333333" 23:21:37 DEBUG "0" 23:21:37 DEBUG }, 23:21:37 DEBUG "2418.2538691883374" 23:21:37 DEBUG } 23:21:37 DEBUG ] 23:21:37 DEBUG ] 23:21:37 DEBUG "value": [ 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG } 23:21:37 DEBUG ] 23:21:37 DEBUG ] 23:21:37 DEBUG "0.021" 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392497" 23:21:37 INFO [http_cmd]: http status code OK 23:21:37 DEBUG --- status code --- 23:21:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:37 DEBUG --- http response --- 23:21:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:37 DEBUG ----- output ----- 23:21:37 DEBUG { 23:21:37 DEBUG "status": "success", 23:21:37 DEBUG "data": { 23:21:37 DEBUG "resultType": "vector", 23:21:37 DEBUG "result": [ 23:21:37 DEBUG { 23:21:37 DEBUG "metric": { 23:21:37 DEBUG "pod": "ds-idrepo-0" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG [...] 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG 23:21:37 DEBUG "pod": "ds-idrepo-2" 23:21:37 DEBUG }, 23:21:37 DEBUG "value": [ 23:21:37 DEBUG 1713392497, 23:21:37 DEBUG "0.06666666666666667" 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG ] 23:21:37 DEBUG } 23:21:37 DEBUG } 23:21:37 INFO 23:21:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:21:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:21:37 INFO [loop_until]: OK (rc = 0) 23:21:37 DEBUG --- stdout --- 23:21:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5435Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 853m 2% 6107Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1166m 3% 11135Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3004Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 3187m 10% 10292Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 4948Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1301m 4% 11534Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 74m 0% 2927Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1292m 4% 9632Mi 8% gke-xlou-cdm-ds-0f601751-j488 722m 2% 5491Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 2929m 9% 11383Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 828m 2% 5966Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 3503m 11% 9740Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 802m 5% 4091Mi 6% 23:21:37 DEBUG --- stderr --- 23:21:37 DEBUG 23:22:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713392554" 23:22:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392554" 23:22:34 ERROR [http_cmd]: ERROR 23:22:34 ERROR --- status code --- 23:22:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:22:34 ERROR --- http response --- 23:22:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:22:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:22:34 INFO [http_cmd]: http status code OK 23:22:34 DEBUG --- status code --- 23:22:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:34 DEBUG --- http response --- 23:22:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:22:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392555" 23:22:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392555" 23:22:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392555" 23:22:35 INFO [http_cmd]: http status code OK 23:22:35 DEBUG --- status code --- 23:22:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:35 DEBUG --- http response --- 23:22:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:22:35 DEBUG ----- output ----- 23:22:35 DEBUG { 23:22:35 DEBUG "status": "success", 23:22:35 DEBUG "data": { 23:22:35 DEBUG "resultType": "vector", 23:22:35 DEBUG "result": [ 23:22:35 DEBUG { 23:22:35 DEBUG "metric": { 23:22:35 DEBUG "pod": "ds-idrepo-0" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG [...] 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 1713392555, 23:22:35 DEBUG "0" 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG } 23:22:35 INFO [http_cmd]: http status code OK 23:22:35 DEBUG --- status code --- 23:22:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:35 DEBUG --- http response --- 23:22:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:35 DEBUG ----- output ----- 23:22:35 DEBUG { 23:22:35 DEBUG "status": "success", 23:22:35 DEBUG "data": { 23:22:35 DEBUG "resultType": "vector", 23:22:35 DEBUG "result": [ 23:22:35 DEBUG { 23:22:35 DEBUG "metric": { 23:22:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG [...] 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 1713392555, 23:22:35 DEBUG "0" 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG } 23:22:35 INFO [http_cmd]: http status code OK 23:22:35 DEBUG --- status code --- 23:22:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:35 DEBUG --- http response --- 23:22:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:35 DEBUG ----- output ----- 23:22:35 DEBUG { 23:22:35 DEBUG "status": "success", 23:22:35 DEBUG "data": { 23:22:35 DEBUG "resultType": "vector", 23:22:35 DEBUG "result": [ 23:22:35 DEBUG { 23:22:35 DEBUG "metric": { 23:22:35 DEBUG "instance": "10.142.0.107:9100" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG [...] 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG "instance": "10.142.0.43:9100" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 1713392555, 23:22:35 DEBUG "0.01353333333333353" 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG } 23:22:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392555" 23:22:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713392555" 23:22:35 INFO [http_cmd]: http status code OK 23:22:35 DEBUG --- status code --- 23:22:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:35 DEBUG --- http response --- 23:22:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:35 DEBUG ----- output ----- 23:22:35 DEBUG { 23:22:35 DEBUG "status": "success", 23:22:35 DEBUG "data": { 23:22:35 DEBUG "resultType": "vector", 23:22:35 DEBUG "result": [ 23:22:35 DEBUG { 23:22:35 DEBUG "metric": { 23:22:35 DEBUG "instance": "10.142.0.107:9100" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG [...] 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG "instance": "10.142.0.43:9100" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 1713392555, 23:22:35 DEBUG "7047971.233333333" 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG } 23:22:35 INFO [http_cmd]: http status code OK 23:22:35 DEBUG --- status code --- 23:22:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:35 DEBUG --- http response --- 23:22:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:35 DEBUG ----- output ----- 23:22:35 DEBUG { 23:22:35 DEBUG "status": "success", 23:22:35 DEBUG "data": { 23:22:35 DEBUG "resultType": "vector", 23:22:35 DEBUG "result": [ 23:22:35 DEBUG { 23:22:35 DEBUG "metric": { 23:22:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG [...] 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 1713392555, 23:22:35 DEBUG "867262464" 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG } 23:22:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392555" 23:22:35 INFO [http_cmd]: http status code OK 23:22:35 DEBUG --- status code --- 23:22:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:35 DEBUG --- http response --- 23:22:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:22:35 DEBUG ----- output ----- 23:22:35 DEBUG { 23:22:35 DEBUG "status": "success", 23:22:35 DEBUG "data": { 23:22:35 DEBUG "resultType": "vector", 23:22:35 DEBUG "result": [ 23:22:35 DEBUG { 23:22:35 DEBUG "metric": { 23:22:35 DEBUG "pod": "ds-cts-0" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG [...] 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG "pod": "ds-cts-2" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 1713392555, 23:22:35 DEBUG "29.966666666666665" 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG } 23:22:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392555" 23:22:35 INFO [http_cmd]: http status code OK 23:22:35 DEBUG --- status code --- 23:22:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:35 DEBUG --- http response --- 23:22:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:35 DEBUG ----- output ----- 23:22:35 DEBUG { 23:22:35 DEBUG "status": "success", 23:22:35 DEBUG "data": { 23:22:35 DEBUG "resultType": "vector", 23:22:35 DEBUG "result": [ 23:22:35 DEBUG { 23:22:35 DEBUG "metric": { 23:22:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG [...] 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG 23:22:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:22:35 DEBUG }, 23:22:35 DEBUG "value": [ 23:22:35 DEBUG 1713392555, 23:22:35 DEBUG "998.4666666666667" 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG ] 23:22:35 DEBUG } 23:22:35 DEBUG } 23:22:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392556" 23:22:36 INFO [http_cmd]: http status code OK 23:22:36 DEBUG --- status code --- 23:22:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:36 DEBUG --- http response --- 23:22:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:22:36 DEBUG ----- output ----- 23:22:36 DEBUG { 23:22:36 DEBUG "status": "success", 23:22:36 DEBUG "data": { 23:22:36 DEBUG "resultType": "vector", 23:22:36 DEBUG "result": [ 23:22:36 DEBUG { 23:22:36 DEBUG "metric": { 23:22:36 DEBUG "pod": "ds-idrepo-0" 23:22:36 DEBUG }, 23:22:36 DEBUG "value": [ 23:22:36 DEBUG 23:22:36 DEBUG 23:22:36 DEBUG 23:22:36 DEBUG [...] 23:22:36 DEBUG 23:22:36 DEBUG 23:22:36 DEBUG 23:22:36 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:22:36 DEBUG }, 23:22:36 DEBUG "value": [ 23:22:36 DEBUG 1713392556, 23:22:36 DEBUG "0" 23:22:36 DEBUG ] 23:22:36 DEBUG } 23:22:36 DEBUG ] 23:22:36 DEBUG } 23:22:36 DEBUG } 23:22:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392556" 23:22:36 INFO [http_cmd]: http status code OK 23:22:36 DEBUG --- status code --- 23:22:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:36 DEBUG --- http response --- 23:22:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:36 DEBUG ----- output ----- 23:22:36 DEBUG { 23:22:36 DEBUG "status": "success", 23:22:36 DEBUG "data": { 23:22:36 DEBUG "resultType": "vector", 23:22:36 DEBUG "result": [ 23:22:36 DEBUG { 23:22:36 DEBUG "metric": { 23:22:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:22:36 DEBUG }, 23:22:36 DEBUG "value": [ 23:22:36 DEBUG 23:22:36 DEBUG 23:22:36 DEBUG 23:22:36 DEBUG [...] 23:22:36 DEBUG 23:22:36 DEBUG 23:22:36 DEBUG 23:22:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:22:36 DEBUG }, 23:22:36 DEBUG "value": [ 23:22:36 DEBUG 1713392556, 23:22:36 DEBUG "0" 23:22:36 DEBUG ] 23:22:36 DEBUG } 23:22:36 DEBUG ] 23:22:36 DEBUG } 23:22:36 DEBUG } 23:22:36 INFO 23:22:36 INFO [loop_until]: kubectl --namespace=xlou top pods 23:22:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:22:36 INFO [loop_until]: OK (rc = 0) 23:22:36 DEBUG --- stdout --- 23:22:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 1146m 8871Mi am-b59c765b7-bgsxp 1074m 7740Mi am-b59c765b7-ch5q4 1146m 8523Mi ds-cts-0 2995m 8009Mi ds-cts-1 2719m 7782Mi ds-cts-2 2926m 8037Mi ds-idrepo-0 473m 3248Mi ds-idrepo-1 322m 2883Mi ds-idrepo-2 315m 2958Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 972Mi idm-b85f9d978-wnhxb 4m 1013Mi lodemon-7cfb6bd874-prp2f 2m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1039m 1332Mi 23:22:36 DEBUG --- stderr --- 23:22:36 DEBUG 23:22:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392556" 23:22:36 INFO [http_cmd]: http status code OK 23:22:36 DEBUG --- status code --- 23:22:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:36 DEBUG --- http response --- 23:22:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:36 DEBUG ----- output ----- 23:22:36 DEBUG { 23:22:36 DEBUG "status": "success", 23:22:36 DEBUG "data": { 23:22:36 DEBUG "resultType": "vector", 23:22:36 DEBUG "result": [ 23:22:36 DEBUG { 23:22:36 DEBUG "metric": { 23:22:36 DEBUG "instance": "10.142.0.107:9100" 23:22:36 DEBUG }, 23:22:36 DEBUG "value": [ 23:22:36 DEBUG 23:22:36 DEBUG 23:22:36 DEBUG 23:22:36 DEBUG [...] 23:22:36 DEBUG 23:22:36 DEBUG 23:22:36 DEBUG 23:22:36 DEBUG "instance": "10.142.0.43:9100" 23:22:36 DEBUG }, 23:22:36 DEBUG "value": [ 23:22:36 DEBUG 1713392556, 23:22:36 DEBUG "256507255808" 23:22:36 DEBUG ] 23:22:36 DEBUG } 23:22:36 DEBUG ] 23:22:36 DEBUG } 23:22:36 DEBUG } 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392557" 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG { 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "data": { 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG "result": [ 23:22:37 DEBUG { 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "pod": "ds-idrepo-0" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG [...] 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG "pod": "ds-idrepo-1" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG "0" 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713392557" 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392557" 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG { 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "data": { 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG "result": [ 23:22:37 DEBUG { 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "pod": "ds-idrepo-0" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG [...] 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG "pod": "ds-cts-2" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG "0" 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG { 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "data": { 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG "result": [ 23:22:37 DEBUG { 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG [...] 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG "0" 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713392557" 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG { 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "data": { 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG "result": [ 23:22:37 DEBUG { 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG [...] 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG "1.0989913908092448" 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392557" 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG { 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "data": { 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG "result": [ 23:22:37 DEBUG { 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG [...] 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG "70336" 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713392557" 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392557" 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392557" 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG { 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "data": { 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG "result": [ 23:22:37 DEBUG { 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG [...] 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG "0.021" 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG { 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "data": { 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG "result": [ 23:22:37 DEBUG { 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "pod": "ds-cts-0" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG [...] 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG "pod": "ds-cts-2" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG "0" 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG { 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "data": { 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG "result": [ 23:22:37 DEBUG { 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG [...] 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG "0" 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392557" 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392557" 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392557" 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392557" 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392557" 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392557" 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG { 23:22:37 DEBUG { 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "data": { 23:22:37 DEBUG "data": { 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG "result": [ 23:22:37 DEBUG "result": [ 23:22:37 DEBUG { 23:22:37 DEBUG { 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "pod": "ds-idrepo-0" 23:22:37 DEBUG "pod": "ds-idrepo-0" 23:22:37 DEBUG }, 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG [...] 23:22:37 DEBUG [...] 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG "pod": "ds-idrepo-2" 23:22:37 DEBUG "pod": "ds-idrepo-2" 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG }, 23:22:37 DEBUG }, 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG "value": [ 23:22:37 DEBUG "value": [ 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 DEBUG 1713392557, 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG "62.9" 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG "8.766666666666666" 23:22:37 DEBUG { 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG ] 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG ] 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG } 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG } 23:22:37 DEBUG "data": { 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:22:37 DEBUG ] 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:37 DEBUG ] 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG } 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG } 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG "result": [ 23:22:37 DEBUG { 23:22:37 DEBUG } 23:22:37 DEBUG { 23:22:37 DEBUG } 23:22:37 DEBUG { 23:22:37 DEBUG { 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "data": { 23:22:37 DEBUG "data": { 23:22:37 DEBUG "data": { 23:22:37 DEBUG "instance": "10.142.0.107:9100" 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG }, 23:22:37 DEBUG "result": [ 23:22:37 DEBUG "result": [ 23:22:37 DEBUG "result": [ 23:22:37 DEBUG "value": [ 23:22:37 DEBUG { 23:22:37 DEBUG { 23:22:37 DEBUG { 23:22:37 DEBUG 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "metric": { 23:22:37 DEBUG 23:22:37 DEBUG "pod": "ds-cts-0" 23:22:37 DEBUG "instance": "10.142.0.107:9100" 23:22:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:22:37 DEBUG 23:22:37 DEBUG }, 23:22:37 DEBUG }, 23:22:37 DEBUG }, 23:22:37 DEBUG [...] 23:22:37 DEBUG "value": [ 23:22:37 DEBUG "value": [ 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG "instance": "10.142.0.43:9100" 23:22:37 DEBUG [...] 23:22:37 DEBUG [...] 23:22:37 DEBUG [...] 23:22:37 DEBUG }, 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG "0.009666666666666488" 23:22:37 DEBUG "pod": "ds-cts-2" 23:22:37 DEBUG "instance": "10.142.0.43:9100" 23:22:37 DEBUG "pod": "ds-idrepo-1" 23:22:37 DEBUG ] 23:22:37 DEBUG }, 23:22:37 DEBUG }, 23:22:37 DEBUG }, 23:22:37 DEBUG } 23:22:37 DEBUG "value": [ 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG ] 23:22:37 DEBUG "value": [ 23:22:37 DEBUG "7045015.366666667" 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG } 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG ] 23:22:37 DEBUG "0.000024326666666666665" 23:22:37 DEBUG } 23:22:37 DEBUG "3602.062760386806" 23:22:37 DEBUG } 23:22:37 DEBUG ] 23:22:37 DEBUG ] 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 DEBUG ] 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392557" 23:22:37 INFO [http_cmd]: http status code OK 23:22:37 DEBUG --- status code --- 23:22:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:37 DEBUG --- http response --- 23:22:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:37 DEBUG ----- output ----- 23:22:37 DEBUG { 23:22:37 DEBUG "status": "success", 23:22:37 DEBUG "data": { 23:22:37 DEBUG "resultType": "vector", 23:22:37 DEBUG "result": [ 23:22:37 DEBUG { 23:22:37 DEBUG "metric": { 23:22:37 DEBUG "pod": "ds-idrepo-0" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG [...] 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG 23:22:37 DEBUG "pod": "ds-idrepo-2" 23:22:37 DEBUG }, 23:22:37 DEBUG "value": [ 23:22:37 DEBUG 1713392557, 23:22:37 DEBUG "6.7" 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG ] 23:22:37 DEBUG } 23:22:37 DEBUG } 23:22:37 INFO 23:22:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:22:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:22:38 INFO [loop_until]: OK (rc = 0) 23:22:38 DEBUG --- stdout --- 23:22:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 5445Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 492m 1% 6138Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1105m 3% 11126Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 78m 0% 2996Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2909m 9% 10299Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 4966Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1215m 4% 11524Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 63m 0% 2910Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 1220m 4% 9626Mi 8% gke-xlou-cdm-ds-0f601751-j488 403m 1% 5527Mi 4% gke-xlou-cdm-ds-0f601751-mc5d 2722m 9% 11395Mi 10% gke-xlou-cdm-ds-0f601751-nj6q 426m 1% 5990Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 3076m 10% 9754Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 1084m 6% 5083Mi 8% 23:22:38 DEBUG --- stderr --- 23:22:38 DEBUG 23:23:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713392614" 23:23:34 ERROR [http_cmd]: ERROR 23:23:34 ERROR --- status code --- 23:23:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:23:34 ERROR --- http response --- 23:23:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:23:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:23:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392614" 23:23:34 INFO [http_cmd]: http status code OK 23:23:34 DEBUG --- status code --- 23:23:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:34 DEBUG --- http response --- 23:23:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:23:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392615" 23:23:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392615" 23:23:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392615" 23:23:35 INFO [http_cmd]: http status code OK 23:23:35 DEBUG --- status code --- 23:23:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:35 DEBUG --- http response --- 23:23:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:23:35 DEBUG ----- output ----- 23:23:35 DEBUG { 23:23:35 DEBUG "status": "success", 23:23:35 DEBUG "data": { 23:23:35 DEBUG "resultType": "vector", 23:23:35 DEBUG "result": [ 23:23:35 DEBUG { 23:23:35 DEBUG "metric": { 23:23:35 DEBUG "pod": "ds-idrepo-0" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG [...] 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG "pod": "am-b59c765b7-7xg4f" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 1713392615, 23:23:35 DEBUG "0.48283186236405917" 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG } 23:23:35 INFO [http_cmd]: http status code OK 23:23:35 DEBUG --- status code --- 23:23:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:35 DEBUG --- http response --- 23:23:35 INFO [http_cmd]: http status code OK 23:23:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:35 DEBUG --- status code --- 23:23:35 DEBUG ----- output ----- 23:23:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:35 DEBUG { 23:23:35 DEBUG --- http response --- 23:23:35 DEBUG "status": "success", 23:23:35 DEBUG "data": { 23:23:35 DEBUG "resultType": "vector", 23:23:35 DEBUG "result": [ 23:23:35 DEBUG { 23:23:35 DEBUG "metric": { 23:23:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG [...] 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 1713392615, 23:23:35 DEBUG "0" 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG } 23:23:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:23:35 DEBUG ----- output ----- 23:23:35 DEBUG { 23:23:35 DEBUG "status": "success", 23:23:35 DEBUG "data": { 23:23:35 DEBUG "resultType": "vector", 23:23:35 DEBUG "result": [ 23:23:35 DEBUG { 23:23:35 DEBUG "metric": { 23:23:35 DEBUG "instance": "10.142.0.107:9100" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG [...] 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG "instance": "10.142.0.43:9100" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 1713392615, 23:23:35 DEBUG "0.01353333333333353" 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG } 23:23:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392615" 23:23:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713392615" 23:23:35 INFO [http_cmd]: http status code OK 23:23:35 DEBUG --- status code --- 23:23:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:35 DEBUG --- http response --- 23:23:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:23:35 DEBUG ----- output ----- 23:23:35 DEBUG { 23:23:35 DEBUG "status": "success", 23:23:35 DEBUG "data": { 23:23:35 DEBUG "resultType": "vector", 23:23:35 DEBUG "result": [ 23:23:35 DEBUG { 23:23:35 DEBUG "metric": { 23:23:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG [...] 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 1713392615, 23:23:35 DEBUG "1487613952" 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG } 23:23:35 INFO [http_cmd]: http status code OK 23:23:35 DEBUG --- status code --- 23:23:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:35 DEBUG --- http response --- 23:23:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:23:35 DEBUG ----- output ----- 23:23:35 DEBUG { 23:23:35 DEBUG "status": "success", 23:23:35 DEBUG "data": { 23:23:35 DEBUG "resultType": "vector", 23:23:35 DEBUG "result": [ 23:23:35 DEBUG { 23:23:35 DEBUG "metric": { 23:23:35 DEBUG "instance": "10.142.0.107:9100" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG [...] 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG "instance": "10.142.0.43:9100" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 1713392615, 23:23:35 DEBUG "7046922.433333334" 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG } 23:23:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392615" 23:23:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392615" 23:23:35 INFO [http_cmd]: http status code OK 23:23:35 DEBUG --- status code --- 23:23:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:35 DEBUG --- http response --- 23:23:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:23:35 DEBUG ----- output ----- 23:23:35 DEBUG { 23:23:35 DEBUG "status": "success", 23:23:35 DEBUG "data": { 23:23:35 DEBUG "resultType": "vector", 23:23:35 DEBUG "result": [ 23:23:35 DEBUG { 23:23:35 DEBUG "metric": { 23:23:35 DEBUG "pod": "ds-cts-0" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG [...] 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG "pod": "ds-cts-2" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 1713392615, 23:23:35 DEBUG "29.9" 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG } 23:23:35 INFO [http_cmd]: http status code OK 23:23:35 DEBUG --- status code --- 23:23:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:35 DEBUG --- http response --- 23:23:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:35 DEBUG ----- output ----- 23:23:35 DEBUG { 23:23:35 DEBUG "status": "success", 23:23:35 DEBUG "data": { 23:23:35 DEBUG "resultType": "vector", 23:23:35 DEBUG "result": [ 23:23:35 DEBUG { 23:23:35 DEBUG "metric": { 23:23:35 DEBUG "pod": "am-b59c765b7-ch5q4" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG [...] 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG 23:23:35 DEBUG "pod": "am-b59c765b7-bgsxp" 23:23:35 DEBUG }, 23:23:35 DEBUG "value": [ 23:23:35 DEBUG 1713392615, 23:23:35 DEBUG "897.1333333333333" 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG ] 23:23:35 DEBUG } 23:23:35 DEBUG } 23:23:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392616" 23:23:36 INFO [http_cmd]: http status code OK 23:23:36 DEBUG --- status code --- 23:23:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:36 DEBUG --- http response --- 23:23:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:23:36 DEBUG ----- output ----- 23:23:36 DEBUG { 23:23:36 DEBUG "status": "success", 23:23:36 DEBUG "data": { 23:23:36 DEBUG "resultType": "vector", 23:23:36 DEBUG "result": [ 23:23:36 DEBUG { 23:23:36 DEBUG "metric": { 23:23:36 DEBUG "pod": "ds-idrepo-0" 23:23:36 DEBUG }, 23:23:36 DEBUG "value": [ 23:23:36 DEBUG 23:23:36 DEBUG 23:23:36 DEBUG 23:23:36 DEBUG [...] 23:23:36 DEBUG 23:23:36 DEBUG 23:23:36 DEBUG 23:23:36 DEBUG "pod": "am-b59c765b7-7xg4f" 23:23:36 DEBUG }, 23:23:36 DEBUG "value": [ 23:23:36 DEBUG 1713392616, 23:23:36 DEBUG "13596.598681372852" 23:23:36 DEBUG ] 23:23:36 DEBUG } 23:23:36 DEBUG ] 23:23:36 DEBUG } 23:23:36 DEBUG } 23:23:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392616" 23:23:36 INFO [http_cmd]: http status code OK 23:23:36 DEBUG --- status code --- 23:23:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:36 DEBUG --- http response --- 23:23:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:36 DEBUG ----- output ----- 23:23:36 DEBUG { 23:23:36 DEBUG "status": "success", 23:23:36 DEBUG "data": { 23:23:36 DEBUG "resultType": "vector", 23:23:36 DEBUG "result": [ 23:23:36 DEBUG { 23:23:36 DEBUG "metric": { 23:23:36 DEBUG "pod": "am-b59c765b7-ch5q4" 23:23:36 DEBUG }, 23:23:36 DEBUG "value": [ 23:23:36 DEBUG 23:23:36 DEBUG 23:23:36 DEBUG 23:23:36 DEBUG [...] 23:23:36 DEBUG 23:23:36 DEBUG 23:23:36 DEBUG 23:23:36 DEBUG "pod": "am-b59c765b7-bgsxp" 23:23:36 DEBUG }, 23:23:36 DEBUG "value": [ 23:23:36 DEBUG 1713392616, 23:23:36 DEBUG "0" 23:23:36 DEBUG ] 23:23:36 DEBUG } 23:23:36 DEBUG ] 23:23:36 DEBUG } 23:23:36 DEBUG } 23:23:36 INFO 23:23:36 INFO [loop_until]: kubectl --namespace=xlou top pods 23:23:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:23:36 INFO [loop_until]: OK (rc = 0) 23:23:36 DEBUG --- stdout --- 23:23:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-b59c765b7-7xg4f 620m 8870Mi am-b59c765b7-bgsxp 643m 7734Mi am-b59c765b7-ch5q4 557m 8514Mi ds-cts-0 2848m 8013Mi ds-cts-1 2189m 7782Mi ds-cts-2 2626m 7983Mi ds-idrepo-0 130m 3248Mi ds-idrepo-1 176m 2883Mi ds-idrepo-2 189m 2957Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 972Mi idm-b85f9d978-wnhxb 4m 1013Mi lodemon-7cfb6bd874-prp2f 2m 64Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 591m 1472Mi 23:23:36 DEBUG --- stderr --- 23:23:36 DEBUG 23:23:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713392616" 23:23:36 INFO [http_cmd]: http status code OK 23:23:36 DEBUG --- status code --- 23:23:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:36 DEBUG --- http response --- 23:23:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:23:36 DEBUG ----- output ----- 23:23:36 DEBUG { 23:23:36 DEBUG "status": "success", 23:23:36 DEBUG "data": { 23:23:36 DEBUG "resultType": "vector", 23:23:36 DEBUG "result": [ 23:23:36 DEBUG { 23:23:36 DEBUG "metric": { 23:23:36 DEBUG "instance": "10.142.0.107:9100" 23:23:36 DEBUG }, 23:23:36 DEBUG "value": [ 23:23:36 DEBUG 23:23:36 DEBUG 23:23:36 DEBUG 23:23:36 DEBUG [...] 23:23:36 DEBUG 23:23:36 DEBUG 23:23:36 DEBUG 23:23:36 DEBUG "instance": "10.142.0.43:9100" 23:23:36 DEBUG }, 23:23:36 DEBUG "value": [ 23:23:36 DEBUG 1713392616, 23:23:36 DEBUG "256653323264" 23:23:36 DEBUG ] 23:23:36 DEBUG } 23:23:36 DEBUG ] 23:23:36 DEBUG } 23:23:36 DEBUG } 23:23:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392617" 23:23:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392617" 23:23:37 INFO [http_cmd]: http status code OK 23:23:37 DEBUG --- status code --- 23:23:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:37 DEBUG --- http response --- 23:23:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:23:37 DEBUG ----- output ----- 23:23:37 DEBUG { 23:23:37 DEBUG "status": "success", 23:23:37 DEBUG "data": { 23:23:37 INFO [http_cmd]: http status code OK 23:23:37 DEBUG "resultType": "vector", 23:23:37 DEBUG --- status code --- 23:23:37 DEBUG "result": [ 23:23:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:37 DEBUG { 23:23:37 DEBUG --- http response --- 23:23:37 DEBUG "metric": { 23:23:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713392617" 23:23:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:23:37 DEBUG ----- output ----- 23:23:37 DEBUG }, 23:23:37 DEBUG { 23:23:37 DEBUG "value": [ 23:23:37 DEBUG "status": "success", 23:23:37 DEBUG 23:23:37 DEBUG "data": { 23:23:37 DEBUG 23:23:37 DEBUG "resultType": "vector", 23:23:37 DEBUG 23:23:37 DEBUG "result": [ 23:23:37 DEBUG [...] 23:23:37 DEBUG { 23:23:37 DEBUG 23:23:37 DEBUG "metric": { 23:23:37 DEBUG 23:23:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:23:37 DEBUG 23:23:37 DEBUG }, 23:23:37 DEBUG "pod": "am-b59c765b7-7xg4f" 23:23:37 DEBUG "value": [ 23:23:37 DEBUG }, 23:23:37 DEBUG 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 23:23:37 DEBUG 1713392617, 23:23:37 DEBUG 23:23:37 DEBUG "0" 23:23:37 DEBUG [...] 23:23:37 DEBUG ] 23:23:37 DEBUG 23:23:37 DEBUG } 23:23:37 DEBUG 23:23:37 DEBUG ] 23:23:37 DEBUG 23:23:37 DEBUG } 23:23:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:23:37 DEBUG } 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 1713392617, 23:23:37 DEBUG "0" 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG } 23:23:37 INFO [http_cmd]: http status code OK 23:23:37 DEBUG --- status code --- 23:23:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:37 DEBUG --- http response --- 23:23:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:23:37 DEBUG ----- output ----- 23:23:37 DEBUG { 23:23:37 DEBUG "status": "success", 23:23:37 DEBUG "data": { 23:23:37 DEBUG "resultType": "vector", 23:23:37 DEBUG "result": [ 23:23:37 DEBUG { 23:23:37 DEBUG "metric": { 23:23:37 DEBUG "pod": "ds-idrepo-0" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG [...] 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG "pod": "ds-cts-2" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 1713392617, 23:23:37 DEBUG "0" 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG } 23:23:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713392617" 23:23:37 INFO [http_cmd]: http status code OK 23:23:37 DEBUG --- status code --- 23:23:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:37 DEBUG --- http response --- 23:23:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:23:37 DEBUG ----- output ----- 23:23:37 DEBUG { 23:23:37 DEBUG "status": "success", 23:23:37 DEBUG "data": { 23:23:37 DEBUG "resultType": "vector", 23:23:37 DEBUG "result": [ 23:23:37 DEBUG { 23:23:37 DEBUG "metric": { 23:23:37 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG [...] 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 1713392617, 23:23:37 DEBUG "0.620421465312598" 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG } 23:23:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392617" 23:23:37 INFO [http_cmd]: http status code OK 23:23:37 DEBUG --- status code --- 23:23:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:37 DEBUG --- http response --- 23:23:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:37 DEBUG ----- output ----- 23:23:37 DEBUG { 23:23:37 DEBUG "status": "success", 23:23:37 DEBUG "data": { 23:23:37 DEBUG "resultType": "vector", 23:23:37 DEBUG "result": [ 23:23:37 DEBUG { 23:23:37 DEBUG "metric": { 23:23:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG [...] 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 1713392617, 23:23:37 DEBUG "13499" 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG } 23:23:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392617" 23:23:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713392617" 23:23:37 INFO [http_cmd]: http status code OK 23:23:37 INFO [http_cmd]: http status code OK 23:23:37 DEBUG --- status code --- 23:23:37 DEBUG --- status code --- 23:23:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:37 DEBUG --- http response --- 23:23:37 DEBUG --- http response --- 23:23:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:37 DEBUG ----- output ----- 23:23:37 DEBUG { 23:23:37 DEBUG "status": "success", 23:23:37 DEBUG "data": { 23:23:37 DEBUG "resultType": "vector", 23:23:37 DEBUG "result": [ 23:23:37 DEBUG { 23:23:37 DEBUG "metric": { 23:23:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG [...] 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 1713392617, 23:23:37 DEBUG "0.021" 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG } 23:23:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:23:37 DEBUG ----- output ----- 23:23:37 DEBUG { 23:23:37 DEBUG "status": "success", 23:23:37 DEBUG "data": { 23:23:37 DEBUG "resultType": "vector", 23:23:37 DEBUG "result": [ 23:23:37 DEBUG { 23:23:37 DEBUG "metric": { 23:23:37 DEBUG "pod": "ds-cts-0" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG [...] 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG "pod": "ds-cts-2" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 1713392617, 23:23:37 DEBUG "0" 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG } 23:23:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392617" 23:23:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392617" 23:23:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392617" 23:23:37 INFO [http_cmd]: http status code OK 23:23:37 DEBUG --- status code --- 23:23:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:37 DEBUG --- http response --- 23:23:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:37 DEBUG ----- output ----- 23:23:37 DEBUG { 23:23:37 DEBUG "status": "success", 23:23:37 DEBUG "data": { 23:23:37 DEBUG "resultType": "vector", 23:23:37 DEBUG "result": [ 23:23:37 DEBUG { 23:23:37 DEBUG "metric": { 23:23:37 DEBUG "pod": "am-b59c765b7-ch5q4" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG [...] 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG "pod": "am-b59c765b7-bgsxp" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 1713392617, 23:23:37 DEBUG "0" 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG } 23:23:37 INFO [http_cmd]: http status code OK 23:23:37 DEBUG --- status code --- 23:23:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:37 DEBUG --- http response --- 23:23:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:37 DEBUG ----- output ----- 23:23:37 DEBUG { 23:23:37 DEBUG "status": "success", 23:23:37 DEBUG "data": { 23:23:37 DEBUG "resultType": "vector", 23:23:37 DEBUG "result": [ 23:23:37 DEBUG { 23:23:37 DEBUG "metric": { 23:23:37 DEBUG "pod": "ds-idrepo-0" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG [...] 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG "pod": "ds-idrepo-2" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 1713392617, 23:23:37 DEBUG "0" 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG } 23:23:37 INFO [http_cmd]: http status code OK 23:23:37 DEBUG --- status code --- 23:23:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:37 DEBUG --- http response --- 23:23:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:23:37 DEBUG ----- output ----- 23:23:37 DEBUG { 23:23:37 DEBUG "status": "success", 23:23:37 DEBUG "data": { 23:23:37 DEBUG "resultType": "vector", 23:23:37 DEBUG "result": [ 23:23:37 DEBUG { 23:23:37 DEBUG "metric": { 23:23:37 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG [...] 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG "pod": "am-b59c765b7-7xg4f" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 1713392617, 23:23:37 DEBUG "0" 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG } 23:23:37 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713392617" 23:23:37 INFO Finished: True 23:23:37 INFO Waiting for threads to register finish flag 23:23:37 INFO [http_cmd]: http status code OK 23:23:37 DEBUG --- status code --- 23:23:37 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:37 DEBUG --- http response --- 23:23:37 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:37 DEBUG ----- output ----- 23:23:37 DEBUG { 23:23:37 DEBUG "status": "success", 23:23:37 DEBUG "data": { 23:23:37 DEBUG "resultType": "vector", 23:23:37 DEBUG "result": [ 23:23:37 DEBUG { 23:23:37 DEBUG "metric": { 23:23:37 DEBUG "pod": "ds-idrepo-0" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG [...] 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG 23:23:37 DEBUG "pod": "ds-idrepo-2" 23:23:37 DEBUG }, 23:23:37 DEBUG "value": [ 23:23:37 DEBUG 1713392617, 23:23:37 DEBUG "0" 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG ] 23:23:37 DEBUG } 23:23:37 DEBUG } 23:24:37 INFO Done. Have a nice day! :) 127.0.0.1 - - [17/Apr/2024 23:24:37] "GET /monitoring/stop HTTP/1.1" 200 - 23:24:40 WARNING File /tmp/lodemon_data-20240417_224222/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6521', ...) 127.0.0.1 - - [17/Apr/2024 23:24:44] "GET /monitoring/process HTTP/1.1" 200 -