==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-56b95674f4-jwfm2 Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Mon, 15 Apr 2024 22:07:45 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=56b95674f4 skaffold.dev/run-id=c7364f1a-aadf-4d24-9cd3-068cc2d322b4 Annotations: Status: Running IP: 10.106.46.144 IPs: IP: 10.106.46.144 Controlled By: ReplicaSet/lodemon-56b95674f4 Containers: lodemon: Container ID: containerd://173c63bd1dc5b9591c0550615e0f4938684652934a0867e1b1483b26b39cde53 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:06c53901427b11990b989f0a16f422c2c89c055d3393605a0d2d532652d23afc Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Mon, 15 Apr 2024 22:07:46 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gwxzz (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-gwxzz: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 42m default-scheduler Successfully assigned xlou/lodemon-56b95674f4-jwfm2 to gke-xlou-cdm-ds-0f601751-mc5d Normal Pulling 42m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 42m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 274.895429ms (274.902074ms including waiting) Normal Created 42m kubelet Created container lodemon Normal Started 42m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.46.144:8080 Press CTRL+C to quit 23:08:52 INFO 23:08:52 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:52 INFO [loop_until]: OK (rc = 0) 23:08:52 DEBUG --- stdout --- 23:08:52 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:52 DEBUG --- stderr --- 23:08:52 DEBUG 23:08:52 INFO 23:08:52 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:52 INFO [loop_until]: OK (rc = 0) 23:08:52 DEBUG --- stdout --- 23:08:52 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:52 DEBUG --- stderr --- 23:08:52 DEBUG 23:08:52 INFO 23:08:52 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:52 INFO [loop_until]: OK (rc = 0) 23:08:52 DEBUG --- stdout --- 23:08:52 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:52 DEBUG --- stderr --- 23:08:52 DEBUG 23:08:52 INFO 23:08:52 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:52 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:52 INFO [loop_until]: OK (rc = 0) 23:08:52 DEBUG --- stdout --- 23:08:52 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:52 DEBUG --- stderr --- 23:08:52 DEBUG 23:08:53 INFO 23:08:53 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:53 INFO [loop_until]: OK (rc = 0) 23:08:53 DEBUG --- stdout --- 23:08:53 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:53 DEBUG --- stderr --- 23:08:53 DEBUG 23:08:53 INFO 23:08:53 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:53 INFO [loop_until]: OK (rc = 0) 23:08:53 DEBUG --- stdout --- 23:08:53 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:53 DEBUG --- stderr --- 23:08:53 DEBUG 23:08:53 INFO 23:08:53 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:53 INFO [loop_until]: OK (rc = 0) 23:08:53 DEBUG --- stdout --- 23:08:53 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:53 DEBUG --- stderr --- 23:08:53 DEBUG 23:08:53 INFO 23:08:53 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:53 INFO [loop_until]: OK (rc = 0) 23:08:53 DEBUG --- stdout --- 23:08:53 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:53 DEBUG --- stderr --- 23:08:53 DEBUG 23:08:53 INFO 23:08:53 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:53 INFO [loop_until]: OK (rc = 0) 23:08:53 DEBUG --- stdout --- 23:08:53 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:53 DEBUG --- stderr --- 23:08:53 DEBUG 23:08:53 INFO 23:08:53 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:53 INFO [loop_until]: OK (rc = 0) 23:08:53 DEBUG --- stdout --- 23:08:53 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:53 DEBUG --- stderr --- 23:08:53 DEBUG 23:08:53 INFO 23:08:53 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:53 INFO [loop_until]: OK (rc = 0) 23:08:53 DEBUG --- stdout --- 23:08:53 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:53 DEBUG --- stderr --- 23:08:53 DEBUG 23:08:53 INFO 23:08:53 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:53 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:54 INFO [loop_until]: OK (rc = 0) 23:08:54 DEBUG --- stdout --- 23:08:54 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:54 DEBUG --- stderr --- 23:08:54 DEBUG 23:08:54 INFO 23:08:54 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:54 INFO [loop_until]: OK (rc = 0) 23:08:54 DEBUG --- stdout --- 23:08:54 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:54 DEBUG --- stderr --- 23:08:54 DEBUG 23:08:54 INFO 23:08:54 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:54 INFO [loop_until]: OK (rc = 0) 23:08:54 DEBUG --- stdout --- 23:08:54 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:54 DEBUG --- stderr --- 23:08:54 DEBUG 23:08:54 INFO 23:08:54 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:54 INFO [loop_until]: OK (rc = 0) 23:08:54 DEBUG --- stdout --- 23:08:54 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:54 DEBUG --- stderr --- 23:08:54 DEBUG 23:08:54 INFO 23:08:54 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:54 INFO [loop_until]: OK (rc = 0) 23:08:54 DEBUG --- stdout --- 23:08:54 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:54 DEBUG --- stderr --- 23:08:54 DEBUG 23:08:54 INFO 23:08:54 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:54 INFO [loop_until]: OK (rc = 0) 23:08:54 DEBUG --- stdout --- 23:08:54 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:54 DEBUG --- stderr --- 23:08:54 DEBUG 23:08:54 INFO 23:08:54 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:54 INFO [loop_until]: OK (rc = 0) 23:08:54 DEBUG --- stdout --- 23:08:54 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:54 DEBUG --- stderr --- 23:08:54 DEBUG 23:08:54 INFO 23:08:54 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:54 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:54 INFO [loop_until]: OK (rc = 0) 23:08:54 DEBUG --- stdout --- 23:08:54 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:54 DEBUG --- stderr --- 23:08:54 DEBUG 23:08:55 INFO 23:08:55 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:55 INFO [loop_until]: OK (rc = 0) 23:08:55 DEBUG --- stdout --- 23:08:55 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:55 DEBUG --- stderr --- 23:08:55 DEBUG 23:08:55 INFO 23:08:55 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:55 INFO [loop_until]: OK (rc = 0) 23:08:55 DEBUG --- stdout --- 23:08:55 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:55 DEBUG --- stderr --- 23:08:55 DEBUG 23:08:55 INFO 23:08:55 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:55 INFO [loop_until]: OK (rc = 0) 23:08:55 DEBUG --- stdout --- 23:08:55 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:55 DEBUG --- stderr --- 23:08:55 DEBUG 23:08:55 INFO 23:08:55 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:55 INFO [loop_until]: OK (rc = 0) 23:08:55 DEBUG --- stdout --- 23:08:55 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:55 DEBUG --- stderr --- 23:08:55 DEBUG 23:08:55 INFO 23:08:55 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:55 INFO [loop_until]: OK (rc = 0) 23:08:55 DEBUG --- stdout --- 23:08:55 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:55 DEBUG --- stderr --- 23:08:55 DEBUG 23:08:55 INFO 23:08:55 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:55 INFO [loop_until]: OK (rc = 0) 23:08:55 DEBUG --- stdout --- 23:08:55 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:55 DEBUG --- stderr --- 23:08:55 DEBUG 23:08:55 INFO 23:08:55 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:55 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:55 INFO [loop_until]: OK (rc = 0) 23:08:55 DEBUG --- stdout --- 23:08:55 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:55 DEBUG --- stderr --- 23:08:55 DEBUG 23:08:56 INFO 23:08:56 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 23:08:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:56 INFO [loop_until]: OK (rc = 0) 23:08:56 DEBUG --- stdout --- 23:08:56 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 23:08:56 DEBUG --- stderr --- 23:08:56 DEBUG 23:08:56 INFO Initializing monitoring instance threads 23:08:56 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 23:08:56 INFO Starting instance threads 23:08:56 INFO 23:08:56 INFO Thread started 23:08:56 INFO [loop_until]: kubectl --namespace=xlou top node 23:08:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:56 INFO 23:08:56 INFO Thread started 23:08:56 INFO [loop_until]: kubectl --namespace=xlou top pods 23:08:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG --- status code --- 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG --- status code --- 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG --- status code --- 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 DEBUG --- status code --- 23:08:56 INFO Thread started 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 DEBUG --- status code --- 23:08:56 INFO Thread started 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:08:56 DEBUG --- http response --- 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG ----- output ----- 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:56 DEBUG ----- output ----- 23:08:56 INFO [loop_until]: OK (rc = 0) 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG { 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:56 DEBUG { 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 DEBUG ----- output ----- 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG { 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG { 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG { 23:08:56 DEBUG --- stdout --- 23:08:56 DEBUG { 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG "status": "success", 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 DEBUG { 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG { 23:08:56 DEBUG { 23:08:56 DEBUG { 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG { 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG { 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8579Mi am-7d597cf7d6-8vr8f 8m 9023Mi am-7d597cf7d6-9wz9h 6m 8906Mi ds-cts-0 13m 8469Mi ds-cts-1 14m 8547Mi ds-cts-2 13m 8692Mi ds-idrepo-0 12m 4977Mi ds-idrepo-1 15m 2693Mi ds-idrepo-2 32m 2698Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 3m 1224Mi lodemon-56b95674f4-jwfm2 1m 54Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 1514Mi 23:08:56 DEBUG "data": { 23:08:56 DEBUG { 23:08:56 DEBUG { 23:08:56 DEBUG "data": { 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 DEBUG "status": "success", 23:08:56 INFO Thread started 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG "data": { 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG "data": { 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:56 DEBUG "status": "success", 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG "data": { 23:08:56 DEBUG "data": { 23:08:56 DEBUG --- stderr --- 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG "data": { 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 DEBUG --- http response --- 23:08:56 INFO Thread started 23:08:56 INFO [loop_until]: OK (rc = 0) 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG "data": { 23:08:56 DEBUG "data": { 23:08:56 DEBUG "data": { 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:56 DEBUG "data": { 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG "data": { 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG 23:08:56 DEBUG "result": [ 23:08:56 DEBUG "data": { 23:08:56 DEBUG "data": { 23:08:56 DEBUG "result": [ 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713218936" 23:08:56 DEBUG --- stdout --- 23:08:56 INFO Thread started 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG "result": [ 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG "result": [ 23:08:56 DEBUG { 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG "result": [ 23:08:56 DEBUG "result": [ 23:08:56 DEBUG { 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG "result": [ 23:08:56 DEBUG { 23:08:56 DEBUG "result": [ 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5074Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 10802Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 52m 0% 10631Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3124Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 88m 0% 5573Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 12869Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2968Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 63m 0% 10502Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 72m 0% 4906Mi 4% gke-xlou-cdm-ds-0f601751-j488 66m 0% 10303Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 2584Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 90m 0% 10510Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 66m 0% 7168Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 68m 0% 5406Mi 9% 23:08:56 DEBUG { 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713218936" 23:08:56 DEBUG { 23:08:56 INFO Thread started 23:08:56 DEBUG "result": [ 23:08:56 DEBUG "result": [ 23:08:56 DEBUG "result": [ 23:08:56 DEBUG { 23:08:56 DEBUG "result": [ 23:08:56 DEBUG { 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG "result": [ 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG --- http response --- 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG { 23:08:56 DEBUG { 23:08:56 DEBUG "metric": { 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG { 23:08:56 DEBUG "metric": { 23:08:56 DEBUG { 23:08:56 DEBUG { 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG --- stderr --- 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG "metric": { 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG { 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218936" 23:08:56 INFO Thread started 23:08:56 DEBUG { 23:08:56 DEBUG { 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG { 23:08:56 DEBUG "metric": { 23:08:56 DEBUG "data": { 23:08:56 DEBUG { 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:56 DEBUG "metric": { 23:08:56 DEBUG "metric": { 23:08:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:08:56 DEBUG "result": [ 23:08:56 DEBUG "metric": { 23:08:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:08:56 DEBUG "metric": { 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG 23:08:56 DEBUG "data": { 23:08:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG "metric": { 23:08:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713218936" 23:08:56 DEBUG "metric": { 23:08:56 INFO Thread started 23:08:56 DEBUG "metric": { 23:08:56 DEBUG "data": { 23:08:56 DEBUG "metric": { 23:08:56 DEBUG "pod": "ds-idrepo-2" 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG "metric": { 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG { 23:08:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:08:56 DEBUG "instance": "10.142.0.107:9100" 23:08:56 DEBUG }, 23:08:56 DEBUG { 23:08:56 DEBUG "pod": "ds-idrepo-2" 23:08:56 DEBUG }, 23:08:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:08:56 DEBUG "data": { 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG }, 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:08:56 DEBUG "instance": "10.142.0.107:9100" 23:08:56 INFO All threads has been started 127.0.0.1 - - [15/Apr/2024 23:08:56] "GET /monitoring/start HTTP/1.1" 200 - 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:08:56 DEBUG }, 23:08:56 DEBUG "result": [ 23:08:56 DEBUG "pod": "ds-idrepo-2" 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG ----- output ----- 23:08:56 ERROR [http_cmd]: ERROR 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG }, 23:08:56 DEBUG }, 23:08:56 DEBUG "value": [ 23:08:56 DEBUG "metric": { 23:08:56 DEBUG }, 23:08:56 DEBUG "value": [ 23:08:56 DEBUG }, 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG "result": [ 23:08:56 DEBUG "value": [ 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG }, 23:08:56 DEBUG }, 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG }, 23:08:56 DEBUG "result": [ 23:08:56 DEBUG }, 23:08:56 DEBUG "value": [ 23:08:56 DEBUG { 23:08:56 DEBUG }, 23:08:56 DEBUG { 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:56 ERROR --- status code --- 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG "data": { 23:08:56 DEBUG "value": [ 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG "result": [ 23:08:56 DEBUG { 23:08:56 DEBUG { 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:56 DEBUG "value": [ 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG "value": [ 23:08:56 DEBUG { 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 23:08:56 DEBUG "metric": { 23:08:56 DEBUG "value": [ 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG ----- output ----- 23:08:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG 23:08:56 DEBUG "instance": "10.142.0.107:9100" 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG { 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG "metric": { 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG 23:08:56 DEBUG "metric": { 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "pod": "ds-idrepo-2" 23:08:56 DEBUG 23:08:56 DEBUG "data": { 23:08:56 DEBUG { 23:08:56 ERROR --- http response --- 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG "result": [ 23:08:56 DEBUG 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "metric": { 23:08:56 DEBUG "data": { 23:08:56 INFO [http_cmd]: http status code OK 23:08:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG { 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG { 23:08:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG [...] 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG [...] 23:08:56 DEBUG 23:08:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG --- status code --- 23:08:56 DEBUG }, 23:08:56 DEBUG [...] 23:08:56 DEBUG 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG [...] 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 23:08:56 DEBUG "result": [ 23:08:56 DEBUG "data": { 23:08:56 DEBUG "metric": { 23:08:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:08:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:08:56 DEBUG [...] 23:08:56 DEBUG 23:08:56 DEBUG [...] 23:08:56 DEBUG 23:08:56 DEBUG [...] 23:08:56 DEBUG 23:08:56 DEBUG [...] 23:08:56 DEBUG }, 23:08:56 DEBUG "result": [ 23:08:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 23:08:56 DEBUG [...] 23:08:56 DEBUG "data": { 23:08:56 DEBUG [...] 23:08:56 DEBUG [...] 23:08:56 DEBUG { 23:08:56 DEBUG "value": [ 23:08:56 DEBUG [...] 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG [...] 23:08:56 DEBUG { 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG "pod": "ds-idrepo-2" 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG { 23:08:56 DEBUG --- http response --- 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG "data": { 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "metric": { 23:08:56 DEBUG "result": [ 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "metric": { 23:08:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "result": [ 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "pod": "ds-idrepo-2" 23:08:56 DEBUG { 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 23:08:56 DEBUG [...] 23:08:56 DEBUG "instance": "10.142.0.43:9100" 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:08:56 DEBUG ----- output ----- 23:08:56 DEBUG 23:08:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:08:56 DEBUG 23:08:56 DEBUG { 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "result": [ 23:08:56 DEBUG 23:08:56 DEBUG "pod": "ds-idrepo-0" 23:08:56 DEBUG [...] 23:08:56 DEBUG 23:08:56 DEBUG }, 23:08:56 DEBUG "metric": { 23:08:56 DEBUG 23:08:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:08:56 DEBUG 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG "pod": "ds-idrepo-0" 23:08:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:08:56 DEBUG 23:08:56 DEBUG }, 23:08:56 DEBUG { 23:08:56 DEBUG [...] 23:08:56 DEBUG }, 23:08:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:08:56 DEBUG "metric": { 23:08:56 DEBUG "instance": "10.142.0.43:9100" 23:08:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:08:56 DEBUG 23:08:56 DEBUG { 23:08:56 DEBUG [...] 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG "pod": "ds-idrepo-0" 23:08:56 DEBUG "value": [ 23:08:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:08:56 DEBUG 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:08:56 DEBUG "0" 23:08:56 DEBUG }, 23:08:56 DEBUG }, 23:08:56 DEBUG [...] 23:08:56 DEBUG "value": [ 23:08:56 DEBUG "status": "success", 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG }, 23:08:56 DEBUG "pod": "ds-idrepo-2" 23:08:56 DEBUG }, 23:08:56 DEBUG }, 23:08:56 DEBUG "value": [ 23:08:56 DEBUG "metric": { 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG }, 23:08:56 DEBUG ] 23:08:56 DEBUG "value": [ 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "data": { 23:08:56 DEBUG 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG "value": [ 23:08:56 DEBUG }, 23:08:56 DEBUG "value": [ 23:08:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG "instance": "10.142.0.107:9100" 23:08:56 DEBUG 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG [...] 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG "instance": "10.142.0.43:9100" 23:08:56 DEBUG "0" 23:08:56 DEBUG "value": [ 23:08:56 DEBUG } 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "resultType": "vector", 23:08:56 DEBUG 23:08:56 DEBUG "0.00014691100126777723" 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG }, 23:08:56 DEBUG "0" 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG "0" 23:08:56 DEBUG "pod": "ds-idrepo-0" 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "0" 23:08:56 DEBUG }, 23:08:56 DEBUG ] 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG ] 23:08:56 DEBUG "0" 23:08:56 DEBUG "1683456000" 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG "result": [ 23:08:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:08:56 DEBUG ] 23:08:56 DEBUG "0" 23:08:56 DEBUG 23:08:56 DEBUG "12141.533333333331" 23:08:56 DEBUG "value": [ 23:08:56 DEBUG ] 23:08:56 DEBUG "value": [ 23:08:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:08:56 DEBUG ] 23:08:56 DEBUG }, 23:08:56 DEBUG "0" 23:08:56 DEBUG [...] 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG ] 23:08:56 DEBUG "value": [ 23:08:56 DEBUG } 23:08:56 DEBUG "0.02257453634225852" 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:08:56 DEBUG [...] 23:08:56 DEBUG { 23:08:56 DEBUG }, 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG 23:08:56 DEBUG ] 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG } 23:08:56 DEBUG 23:08:56 DEBUG }, 23:08:56 DEBUG } 23:08:56 DEBUG "value": [ 23:08:56 DEBUG ] 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG } 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG ] 23:08:56 DEBUG ] 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG "metric": { 23:08:56 DEBUG "value": [ 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG 23:08:56 DEBUG } 23:08:56 DEBUG "0" 23:08:56 DEBUG ] 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG ] 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG } 23:08:56 DEBUG 23:08:56 DEBUG [...] 23:08:56 DEBUG "pod": "ds-idrepo-0" 23:08:56 DEBUG ] 23:08:56 DEBUG "5471.433333333333" 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 23:08:56 DEBUG "instance": "10.142.0.107:9100" 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG [...] 23:08:56 DEBUG ] 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG } 23:08:56 DEBUG "0" 23:08:56 DEBUG ] 23:08:56 DEBUG 23:08:56 DEBUG 23:08:56 DEBUG }, 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG 23:08:56 DEBUG }, 23:08:56 DEBUG "0" 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG [...] 23:08:56 DEBUG "0" 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG "pod": "ds-idrepo-0" 23:08:56 DEBUG 23:08:56 DEBUG "value": [ 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG "0" 23:08:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:08:56 DEBUG "value": [ 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG }, 23:08:56 DEBUG 23:08:56 DEBUG } 23:08:56 DEBUG 23:08:56 DEBUG } 23:08:56 DEBUG 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG "0" 23:08:56 DEBUG }, 23:08:56 DEBUG } 23:08:56 DEBUG "value": [ 23:08:56 DEBUG 23:08:56 DEBUG ] 23:08:56 DEBUG "pod": "ds-cts-2" 23:08:56 DEBUG 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG 23:08:56 DEBUG } 23:08:56 DEBUG }, 23:08:56 DEBUG "instance": "10.142.0.43:9100" 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG "0" 23:08:56 DEBUG "value": [ 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG "1" 23:08:56 DEBUG [...] 23:08:56 DEBUG } 23:08:56 DEBUG "value": [ 23:08:56 DEBUG }, 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG "value": [ 23:08:56 DEBUG } 23:08:56 DEBUG "0.08" 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG 23:08:56 DEBUG "0" 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG ] 23:08:56 DEBUG ] 23:08:56 DEBUG ] 23:08:56 DEBUG 23:08:56 DEBUG ] 23:08:56 DEBUG "13893071872" 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG "instance": "10.142.0.43:9100" 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG }, 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG "value": [ 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG 1713218936, 23:08:56 DEBUG } 23:08:56 DEBUG } 23:08:56 DEBUG "0.0005333333333335114" 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG ] 23:08:56 DEBUG } 23:08:56 DEBUG } 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713218996" 23:09:56 INFO [loop_until]: kubectl --namespace=xlou top pods 23:09:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO [loop_until]: kubectl --namespace=xlou top node 23:09:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713218996" 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG --- status code --- 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG --- status code --- 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG --- status code --- 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG --- http response --- 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG { 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:56 DEBUG ----- output ----- 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 ERROR [http_cmd]: ERROR 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 DEBUG ----- output ----- 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG { 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG { 23:09:56 ERROR --- status code --- 23:09:56 DEBUG ----- output ----- 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713218996" 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG { 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG { 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG "data": { 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 DEBUG { 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG { 23:09:56 DEBUG "status": "success", 23:09:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:09:56 DEBUG { 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713218996" 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713218996" 23:09:56 DEBUG --- status code --- 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG "data": { 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 INFO [loop_until]: OK (rc = 0) 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG "data": { 23:09:56 ERROR --- http response --- 23:09:56 DEBUG "status": "success", 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG "data": { 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG --- http response --- 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG "data": { 23:09:56 DEBUG "result": [ 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG "data": { 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG --- stdout --- 23:09:56 DEBUG "data": { 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:09:56 DEBUG "data": { 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:09:56 DEBUG --- status code --- 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG { 23:09:56 DEBUG { 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG "result": [ 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 127m 8581Mi am-7d597cf7d6-8vr8f 8m 9023Mi am-7d597cf7d6-9wz9h 14m 8908Mi ds-cts-0 13m 8468Mi ds-cts-1 13m 8547Mi ds-cts-2 24m 8693Mi ds-idrepo-0 107m 4976Mi ds-idrepo-1 44m 2692Mi ds-idrepo-2 27m 2698Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 3m 1224Mi lodemon-56b95674f4-jwfm2 61m 61Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1961m 1920Mi 23:09:56 DEBUG "result": [ 23:09:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG "result": [ 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG "result": [ 23:09:56 DEBUG "metric": { 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG "result": [ 23:09:56 DEBUG { 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG "result": [ 23:09:56 DEBUG --- stderr --- 23:09:56 INFO [http_cmd]: http status code OK 23:09:56 DEBUG { 23:09:56 DEBUG "result": [ 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:56 DEBUG { 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG { 23:09:56 DEBUG "instance": "10.142.0.107:9100" 23:09:56 DEBUG "data": { 23:09:56 DEBUG { 23:09:56 DEBUG "metric": { 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:56 DEBUG { 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:56 DEBUG 23:09:56 DEBUG --- status code --- 23:09:56 DEBUG "metric": { 23:09:56 DEBUG { 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:56 DEBUG { 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG "metric": { 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG "metric": { 23:09:56 DEBUG }, 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG "metric": { 23:09:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG "metric": { 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:56 DEBUG --- http response --- 23:09:56 DEBUG "instance": "10.142.0.107:9100" 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG { 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG { 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG { 23:09:56 DEBUG { 23:09:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "result": [ 23:09:56 DEBUG "pod": "ds-idrepo-2" 23:09:56 DEBUG }, 23:09:56 DEBUG { 23:09:56 DEBUG "pod": "ds-idrepo-2" 23:09:56 DEBUG { 23:09:56 DEBUG "metric": { 23:09:56 DEBUG }, 23:09:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:56 DEBUG { 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG { 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG { 23:09:56 DEBUG "data": { 23:09:56 DEBUG { 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG }, 23:09:56 DEBUG { 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG { 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG { 23:09:56 DEBUG }, 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG }, 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG "pod": "ds-idrepo-2" 23:09:56 DEBUG "value": [ 23:09:56 DEBUG ----- output ----- 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG "data": { 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG "data": { 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG "data": { 23:09:56 DEBUG "data": { 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG { 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG { 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG 23:09:56 DEBUG "metric": { 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 23:09:56 DEBUG "data": { 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "data": { 23:09:56 DEBUG }, 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 23:09:56 DEBUG "data": { 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG "data": { 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG "data": { 23:09:56 DEBUG "result": [ 23:09:56 DEBUG "data": { 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG 23:09:56 DEBUG "data": { 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG "data": { 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "data": { 23:09:56 DEBUG 23:09:56 DEBUG "instance": "10.142.0.107:9100" 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG 23:09:56 DEBUG { 23:09:56 DEBUG 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG "result": [ 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG "result": [ 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG { 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG "result": [ 23:09:56 DEBUG { 23:09:56 DEBUG 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG "data": { 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG [...] 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "result": [ 23:09:56 DEBUG { 23:09:56 DEBUG "result": [ 23:09:56 DEBUG 23:09:56 DEBUG "status": "success", 23:09:56 DEBUG 23:09:56 DEBUG "result": [ 23:09:56 DEBUG { 23:09:56 DEBUG "result": [ 23:09:56 DEBUG { 23:09:56 DEBUG "result": [ 23:09:56 DEBUG "metric": { 23:09:56 DEBUG "result": [ 23:09:56 DEBUG "result": [ 23:09:56 DEBUG "metric": { 23:09:56 DEBUG "pod": "ds-idrepo-2" 23:09:56 DEBUG "result": [ 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG "result": [ 23:09:56 DEBUG 23:09:56 DEBUG "result": [ 23:09:56 DEBUG 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG { 23:09:56 DEBUG 23:09:56 DEBUG "data": { 23:09:56 DEBUG "resultType": "vector", 23:09:56 DEBUG [...] 23:09:56 DEBUG "metric": { 23:09:56 DEBUG { 23:09:56 DEBUG "metric": { 23:09:56 DEBUG { 23:09:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:09:56 DEBUG { 23:09:56 DEBUG { 23:09:56 DEBUG 23:09:56 DEBUG }, 23:09:56 DEBUG { 23:09:56 DEBUG "result": [ 23:09:56 DEBUG { 23:09:56 DEBUG 23:09:56 DEBUG { 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG "metric": { 23:09:56 DEBUG "metric": { 23:09:56 DEBUG [...] 23:09:56 DEBUG { 23:09:56 DEBUG "result": [ 23:09:56 DEBUG 23:09:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:09:56 DEBUG "metric": { 23:09:56 DEBUG "instance": "10.142.0.107:9100" 23:09:56 DEBUG "metric": { 23:09:56 DEBUG }, 23:09:56 DEBUG "metric": { 23:09:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:09:56 DEBUG [...] 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "metric": { 23:09:56 DEBUG { 23:09:56 DEBUG "metric": { 23:09:56 DEBUG [...] 23:09:56 DEBUG "metric": { 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:09:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:09:56 DEBUG 23:09:56 DEBUG "metric": { 23:09:56 DEBUG { 23:09:56 DEBUG 23:09:56 DEBUG }, 23:09:56 DEBUG "pod": "ds-idrepo-2" 23:09:56 DEBUG }, 23:09:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "metric": { 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:09:56 DEBUG "metric": { 23:09:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:09:56 DEBUG 23:09:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG }, 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:09:56 DEBUG "metric": { 23:09:56 DEBUG 23:09:56 DEBUG "value": [ 23:09:56 DEBUG }, 23:09:56 DEBUG "value": [ 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG "pod": "ds-idrepo-2" 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG }, 23:09:56 DEBUG "value": [ 23:09:56 DEBUG }, 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "instance": "10.142.0.43:9100" 23:09:56 DEBUG "value": [ 23:09:56 DEBUG [...] 23:09:56 DEBUG 23:09:56 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:09:56 DEBUG 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 23:09:56 DEBUG }, 23:09:56 DEBUG "instance": "10.142.0.107:9100" 23:09:56 DEBUG "pod": "ds-cts-2" 23:09:56 DEBUG 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "pod": "ds-idrepo-2" 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "pod": "ds-idrepo-0" 23:09:56 DEBUG }, 23:09:56 DEBUG "pod": "ds-idrepo-0" 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "instance": "10.142.0.43:9100" 23:09:56 DEBUG "value": [ 23:09:56 DEBUG }, 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:09:56 DEBUG [...] 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG }, 23:09:56 DEBUG "value": [ 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG [...] 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 23:09:56 DEBUG }, 23:09:56 DEBUG [...] 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "0" 23:09:56 DEBUG ] 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:09:56 DEBUG }, 23:09:56 DEBUG "instance": "10.142.0.43:9100" 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG "0" 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG [...] 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "pod": "ds-idrepo-0" 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "12596.133333333331" 23:09:56 DEBUG "value": [ 23:09:56 DEBUG }, 23:09:56 DEBUG "0" 23:09:56 DEBUG ] 23:09:56 DEBUG "0" 23:09:56 DEBUG 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG } 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:09:56 DEBUG [...] 23:09:56 DEBUG ] 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG "value": [ 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG 23:09:56 DEBUG "0" 23:09:56 DEBUG ] 23:09:56 DEBUG 23:09:56 DEBUG [...] 23:09:56 DEBUG 23:09:56 DEBUG ] 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "instance": "10.142.0.43:9100" 23:09:56 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:09:56 DEBUG 23:09:56 DEBUG "0.08" 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG } 23:09:56 DEBUG "0" 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG 23:09:56 DEBUG } 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:09:56 DEBUG }, 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG ] 23:09:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "0" 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 23:09:56 DEBUG ] 23:09:56 DEBUG ] 23:09:56 DEBUG "13895773184" 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:09:56 DEBUG ] 23:09:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:09:56 DEBUG 23:09:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "pod": "ds-idrepo-0" 23:09:56 DEBUG } 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG 23:09:56 DEBUG ] 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG }, 23:09:56 DEBUG } 23:09:56 DEBUG }, 23:09:56 DEBUG 23:09:56 DEBUG }, 23:09:56 DEBUG "instance": "10.142.0.43:9100" 23:09:56 DEBUG }, 23:09:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG }, 23:09:56 DEBUG ] 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:09:56 DEBUG "pod": "ds-idrepo-0" 23:09:56 DEBUG } 23:09:56 DEBUG "0" 23:09:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG "value": [ 23:09:56 DEBUG } 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG "value": [ 23:09:56 DEBUG }, 23:09:56 DEBUG "value": [ 23:09:56 DEBUG }, 23:09:56 DEBUG "3258.4666666666662" 23:09:56 DEBUG "984.6812137951042" 23:09:56 DEBUG "value": [ 23:09:56 DEBUG } 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG }, 23:09:56 DEBUG }, 23:09:56 DEBUG ] 23:09:56 DEBUG ] 23:09:56 DEBUG }, 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG 23:09:56 DEBUG "996.2" 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG "value": [ 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG "value": [ 23:09:56 DEBUG ] 23:09:56 DEBUG ] 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG } 23:09:56 DEBUG "2109165568" 23:09:56 DEBUG "value": [ 23:09:56 DEBUG "value": [ 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG "value": [ 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG "995.6" 23:09:56 DEBUG "pod": "ds-idrepo-0" 23:09:56 DEBUG }, 23:09:56 DEBUG ] 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG "2.0070440351946366" 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG "0" 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG "0" 23:09:56 DEBUG "value": [ 23:09:56 DEBUG } 23:09:56 DEBUG "0.0006000000000000227" 23:09:56 DEBUG ] 23:09:56 DEBUG "0" 23:09:56 DEBUG ] 23:09:56 DEBUG ] 23:09:56 DEBUG ] 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG "996.0666666666666" 23:09:56 DEBUG "0" 23:09:56 DEBUG } 23:09:56 DEBUG "14927.5" 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG 1713218996, 23:09:56 DEBUG ] 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG ] 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG "0" 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG ] 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 DEBUG } 23:09:56 INFO [loop_until]: OK (rc = 0) 23:09:56 DEBUG --- stdout --- 23:09:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 5072Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 10808Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 178m 0% 10628Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3127Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 94m 0% 5575Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 253m 0% 12853Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2973Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 92m 0% 10507Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 86m 0% 4913Mi 4% gke-xlou-cdm-ds-0f601751-j488 64m 0% 10304Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 195m 0% 2428Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 82m 0% 10521Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 62m 0% 7160Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 1892m 11% 5677Mi 9% 23:09:56 DEBUG --- stderr --- 23:09:56 DEBUG 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219056" 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219056" 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 ERROR [http_cmd]: ERROR 23:10:56 ERROR --- status code --- 23:10:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:10:56 ERROR --- http response --- 23:10:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:10:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219056" 23:10:56 INFO 23:10:56 INFO [loop_until]: kubectl --namespace=xlou top pods 23:10:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219056" 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219056" 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG --- status code --- 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:56 DEBUG --- http response --- 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG ----- output ----- 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG { 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG { 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG { 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG { 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG "data": { 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG --- http response --- 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG "data": { 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG "data": { 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "data": { 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219056" 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:56 DEBUG "result": [ 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG { 23:10:56 DEBUG "result": [ 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG "result": [ 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG { 23:10:56 DEBUG "result": [ 23:10:56 DEBUG { 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG "data": { 23:10:56 DEBUG { 23:10:56 DEBUG { 23:10:56 DEBUG { 23:10:56 DEBUG "metric": { 23:10:56 DEBUG { 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG { 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "data": { 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "result": [ 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG "data": { 23:10:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:10:56 DEBUG "data": { 23:10:56 DEBUG }, 23:10:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "instance": "10.142.0.107:9100" 23:10:56 DEBUG { 23:10:56 DEBUG { 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG }, 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "value": [ 23:10:56 DEBUG }, 23:10:56 DEBUG "result": [ 23:10:56 DEBUG }, 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG "result": [ 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "result": [ 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG { 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "pod": "ds-idrepo-2" 23:10:56 DEBUG "data": { 23:10:56 DEBUG { 23:10:56 DEBUG 23:10:56 DEBUG { 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "instance": "10.142.0.107:9100" 23:10:56 DEBUG }, 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "metric": { 23:10:56 DEBUG 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "instance": "10.142.0.107:9100" 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG }, 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "result": [ 23:10:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG }, 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 23:10:56 DEBUG { 23:10:56 DEBUG }, 23:10:56 DEBUG [...] 23:10:56 DEBUG [...] 23:10:56 DEBUG "value": [ 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG [...] 23:10:56 DEBUG }, 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 23:10:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:10:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:10:56 DEBUG [...] 23:10:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG [...] 23:10:56 DEBUG }, 23:10:56 DEBUG }, 23:10:56 DEBUG 23:10:56 DEBUG }, 23:10:56 DEBUG "instance": "10.142.0.43:9100" 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG }, 23:10:56 DEBUG 23:10:56 DEBUG "pod": "ds-idrepo-0" 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "instance": "10.142.0.43:9100" 23:10:56 DEBUG }, 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG "0" 23:10:56 DEBUG "2164842496" 23:10:56 DEBUG "instance": "10.142.0.43:9100" 23:10:56 DEBUG "2302.4333333333334" 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG }, 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 23:10:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:10:56 DEBUG ] 23:10:56 DEBUG ] 23:10:56 DEBUG }, 23:10:56 DEBUG ] 23:10:56 DEBUG "12008.866666666665" 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG 23:10:56 DEBUG }, 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG "value": [ 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG "0" 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG ] 23:10:56 DEBUG ] 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG "0.000500000000000019" 23:10:56 DEBUG ] 23:10:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG "3109.7666666666664" 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG }, 23:10:56 DEBUG "0" 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG "value": [ 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG "9.578709846931643" 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 DEBUG } 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219056" 23:10:56 DEBUG } 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG ] 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:56 DEBUG ----- output ----- 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 DEBUG } 23:10:56 DEBUG { 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219056" 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG --- status code --- 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 INFO [loop_until]: OK (rc = 0) 23:10:56 DEBUG } 23:10:56 DEBUG "status": "success", 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219056" 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG --- status code --- 23:10:56 INFO 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG --- stdout --- 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG "data": { 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG --- status code --- 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG --- status code --- 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 INFO [loop_until]: kubectl --namespace=xlou top node 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 9783m 8589Mi am-7d597cf7d6-8vr8f 7728m 7626Mi am-7d597cf7d6-9wz9h 12584m 8924Mi ds-cts-0 6039m 8480Mi ds-cts-1 4524m 8586Mi ds-cts-2 4283m 8705Mi ds-idrepo-0 11065m 4977Mi ds-idrepo-1 1597m 2693Mi ds-idrepo-2 1638m 2699Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 3m 1224Mi lodemon-56b95674f4-jwfm2 10m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6771m 1964Mi 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG --- http response --- 23:10:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG --- stderr --- 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG "result": [ 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG { 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 INFO [http_cmd]: http status code OK 23:10:56 DEBUG { 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:56 DEBUG "metric": { 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG { 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG --- status code --- 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG { 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG "instance": "10.142.0.107:9100" 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG { 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:56 DEBUG "data": { 23:10:56 DEBUG { 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG { 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG { 23:10:56 DEBUG { 23:10:56 DEBUG }, 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:56 DEBUG "data": { 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG --- http response --- 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG { 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG "data": { 23:10:56 DEBUG { 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG "value": [ 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "data": { 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:56 DEBUG { 23:10:56 DEBUG { 23:10:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:56 DEBUG "result": [ 23:10:56 DEBUG "data": { 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG "data": { 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG "data": { 23:10:56 DEBUG "data": { 23:10:56 DEBUG 23:10:56 DEBUG { 23:10:56 DEBUG "result": [ 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG ----- output ----- 23:10:56 DEBUG { 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "data": { 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "result": [ 23:10:56 DEBUG "data": { 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG { 23:10:56 DEBUG "result": [ 23:10:56 DEBUG { 23:10:56 DEBUG "data": { 23:10:56 DEBUG "data": { 23:10:56 DEBUG { 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "result": [ 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "result": [ 23:10:56 DEBUG { 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "result": [ 23:10:56 DEBUG "result": [ 23:10:56 DEBUG 23:10:56 DEBUG "data": { 23:10:56 DEBUG "metric": { 23:10:56 DEBUG { 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "result": [ 23:10:56 DEBUG "status": "success", 23:10:56 DEBUG "pod": "ds-idrepo-2" 23:10:56 DEBUG { 23:10:56 DEBUG "result": [ 23:10:56 DEBUG { 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "result": [ 23:10:56 DEBUG { 23:10:56 DEBUG { 23:10:56 DEBUG [...] 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "pod": "ds-idrepo-2" 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "data": { 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG { 23:10:56 DEBUG "data": { 23:10:56 DEBUG }, 23:10:56 DEBUG "metric": { 23:10:56 DEBUG { 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "pod": "ds-idrepo-2" 23:10:56 DEBUG { 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "metric": { 23:10:56 DEBUG 23:10:56 DEBUG "result": [ 23:10:56 DEBUG }, 23:10:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "result": [ 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "resultType": "vector", 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "instance": "10.142.0.107:9100" 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:10:56 DEBUG }, 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:10:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:10:56 DEBUG 23:10:56 DEBUG { 23:10:56 DEBUG "value": [ 23:10:56 DEBUG }, 23:10:56 DEBUG "result": [ 23:10:56 DEBUG { 23:10:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:10:56 DEBUG "result": [ 23:10:56 DEBUG 23:10:56 DEBUG }, 23:10:56 DEBUG "pod": "ds-idrepo-2" 23:10:56 DEBUG }, 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:10:56 DEBUG }, 23:10:56 DEBUG }, 23:10:56 DEBUG 23:10:56 DEBUG "metric": { 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG { 23:10:56 DEBUG "metric": { 23:10:56 DEBUG }, 23:10:56 DEBUG { 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG }, 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 23:10:56 DEBUG }, 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "instance": "10.142.0.43:9100" 23:10:56 DEBUG "pod": "ds-idrepo-2" 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG "metric": { 23:10:56 DEBUG "pod": "ds-idrepo-2" 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "metric": { 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG }, 23:10:56 DEBUG }, 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:10:56 DEBUG }, 23:10:56 DEBUG 23:10:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "value": [ 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG }, 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 23:10:56 DEBUG }, 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG [...] 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG [...] 23:10:56 DEBUG [...] 23:10:56 DEBUG "0" 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG ] 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG } 23:10:56 DEBUG [...] 23:10:56 DEBUG "pod": "ds-idrepo-0" 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG "pod": "ds-cts-2" 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG "instance": "10.142.0.43:9100" 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG }, 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG ] 23:10:56 DEBUG 23:10:56 DEBUG }, 23:10:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:10:56 DEBUG [...] 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG }, 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG }, 23:10:56 DEBUG "pod": "ds-idrepo-0" 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:10:56 DEBUG } 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG }, 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG 23:10:56 DEBUG "value": [ 23:10:56 DEBUG }, 23:10:56 DEBUG 23:10:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:10:56 DEBUG "0" 23:10:56 DEBUG }, 23:10:56 DEBUG } 23:10:56 DEBUG 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 23:10:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:10:56 DEBUG }, 23:10:56 DEBUG 23:10:56 DEBUG "0" 23:10:56 DEBUG "pod": "ds-idrepo-0" 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:10:56 DEBUG }, 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG "pod": "ds-idrepo-0" 23:10:56 DEBUG "0" 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG 23:10:56 DEBUG "pod": "ds-idrepo-0" 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 23:10:56 DEBUG ] 23:10:56 DEBUG }, 23:10:56 DEBUG "13897956352" 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG }, 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG "2301.5" 23:10:56 DEBUG }, 23:10:56 DEBUG ] 23:10:56 DEBUG "0.08" 23:10:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:10:56 DEBUG }, 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:10:56 DEBUG } 23:10:56 DEBUG "value": [ 23:10:56 DEBUG ] 23:10:56 DEBUG "0" 23:10:56 DEBUG ] 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "6.72557569363775" 23:10:56 DEBUG ] 23:10:56 DEBUG "value": [ 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG }, 23:10:56 DEBUG "value": [ 23:10:56 DEBUG "0" 23:10:56 DEBUG }, 23:10:56 DEBUG ] 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG "value": [ 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG ] 23:10:56 DEBUG "value": [ 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG "0" 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG "0" 23:10:56 DEBUG } 23:10:56 DEBUG 1713219056, 23:10:56 DEBUG "0" 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG ] 23:10:56 DEBUG "363017.08985094563" 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG "2303.1" 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG "133640.5" 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG ] 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG ] 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 DEBUG } 23:10:56 INFO [loop_until]: OK (rc = 0) 23:10:56 DEBUG --- stdout --- 23:10:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5075Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4474m 14% 10888Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10198m 34% 10640Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17471m 58% 3057Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1828m 6% 5583Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 6468m 21% 12863Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 67m 0% 2989Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 7931m 26% 9116Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1681m 5% 4948Mi 4% gke-xlou-cdm-ds-0f601751-j488 4273m 14% 10372Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 73m 0% 2455Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 13421m 44% 10550Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12045m 40% 7154Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6885m 43% 5850Mi 9% 23:10:56 DEBUG --- stderr --- 23:10:56 DEBUG 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 ERROR [http_cmd]: ERROR 23:11:56 ERROR --- status code --- 23:11:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:11:56 ERROR --- http response --- 23:11:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:11:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "result": [ 23:11:56 DEBUG { 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "instance": "10.142.0.107:9100" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "instance": "10.142.0.43:9100" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "0.0004666666666665264" 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- status code --- 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:11:56 DEBUG { 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "resultType": "vector", 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219116" 23:11:56 DEBUG "data": { 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "result": [ 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG "result": [ 23:11:56 DEBUG "result": [ 23:11:56 DEBUG { 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG { 23:11:56 DEBUG { 23:11:56 DEBUG "metric": { 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "pod": "ds-idrepo-2" 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:11:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:11:56 DEBUG }, 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:56 DEBUG }, 23:11:56 DEBUG }, 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:56 DEBUG "value": [ 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG "value": [ 23:11:56 DEBUG "value": [ 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG 23:11:56 DEBUG { 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG { 23:11:56 DEBUG 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG 23:11:56 DEBUG "data": { 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "data": { 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:56 DEBUG [...] 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG [...] 23:11:56 DEBUG [...] 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG 23:11:56 DEBUG "result": [ 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "result": [ 23:11:56 DEBUG { 23:11:56 DEBUG 23:11:56 DEBUG { 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG 23:11:56 DEBUG "metric": { 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "data": { 23:11:56 DEBUG "pod": "ds-idrepo-0" 23:11:56 DEBUG "instance": "10.142.0.107:9100" 23:11:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:11:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:11:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG }, 23:11:56 DEBUG }, 23:11:56 DEBUG }, 23:11:56 DEBUG }, 23:11:56 DEBUG }, 23:11:56 DEBUG "result": [ 23:11:56 DEBUG "value": [ 23:11:56 DEBUG "value": [ 23:11:56 DEBUG "value": [ 23:11:56 DEBUG "value": [ 23:11:56 DEBUG "value": [ 23:11:56 DEBUG { 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "0" 23:11:56 DEBUG 23:11:56 DEBUG "0" 23:11:56 DEBUG "0" 23:11:56 DEBUG 23:11:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:11:56 DEBUG ] 23:11:56 DEBUG 23:11:56 DEBUG ] 23:11:56 DEBUG ] 23:11:56 DEBUG 23:11:56 DEBUG }, 23:11:56 DEBUG } 23:11:56 DEBUG [...] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG [...] 23:11:56 DEBUG "value": [ 23:11:56 DEBUG ] 23:11:56 DEBUG 23:11:56 DEBUG ] 23:11:56 DEBUG ] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG } 23:11:56 DEBUG 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG } 23:11:56 DEBUG 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "instance": "10.142.0.43:9100" 23:11:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:11:56 DEBUG [...] 23:11:56 DEBUG }, 23:11:56 DEBUG }, 23:11:56 DEBUG 23:11:56 DEBUG "value": [ 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:11:56 DEBUG "2297.733333333333" 23:11:56 DEBUG "4500.066666666667" 23:11:56 DEBUG }, 23:11:56 DEBUG ] 23:11:56 DEBUG ] 23:11:56 DEBUG "value": [ 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG ] 23:11:56 DEBUG ] 23:11:56 DEBUG "6.665973800970337" 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "result": [ 23:11:56 DEBUG { 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "2158505984" 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "result": [ 23:11:56 DEBUG { 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "instance": "10.142.0.107:9100" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "instance": "10.142.0.43:9100" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "12328.066666666664" 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "result": [ 23:11:56 DEBUG { 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "pod": "ds-idrepo-2" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "pod": "ds-idrepo-0" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "0" 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 INFO 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [loop_until]: kubectl --namespace=xlou top pods 23:11:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "result": [ 23:11:56 DEBUG { 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "pod": "ds-idrepo-2" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "pod": "ds-idrepo-0" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "0" 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "result": [ 23:11:56 DEBUG { 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "pod": "ds-idrepo-2" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "pod": "ds-idrepo-0" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "0" 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219116" 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "result": [ 23:11:56 DEBUG { 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "pod": "ds-idrepo-2" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "pod": "ds-cts-2" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "0" 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219116" 23:11:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219116" 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 INFO [loop_until]: OK (rc = 0) 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- status code --- 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- stdout --- 23:11:56 DEBUG --- status code --- 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 9947m 8590Mi am-7d597cf7d6-8vr8f 7900m 7626Mi am-7d597cf7d6-9wz9h 12616m 8927Mi ds-cts-0 6184m 8517Mi ds-cts-1 4422m 8615Mi ds-cts-2 4352m 8746Mi ds-idrepo-0 12090m 4977Mi ds-idrepo-1 1665m 2693Mi ds-idrepo-2 1649m 2699Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 3m 1224Mi lodemon-56b95674f4-jwfm2 9m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6491m 1968Mi 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- http response --- 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG --- stderr --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:56 DEBUG 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:56 INFO [http_cmd]: http status code OK 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG { 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:56 DEBUG { 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:56 DEBUG --- status code --- 23:11:56 DEBUG { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG { 23:11:56 DEBUG { 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG { 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "data": { 23:11:56 DEBUG { 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG { 23:11:56 DEBUG "data": { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG { 23:11:56 DEBUG --- http response --- 23:11:56 DEBUG "data": { 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "data": { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:56 DEBUG "result": [ 23:11:56 DEBUG "data": { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "result": [ 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "result": [ 23:11:56 DEBUG ----- output ----- 23:11:56 DEBUG { 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "data": { 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "result": [ 23:11:56 DEBUG "result": [ 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG { 23:11:56 DEBUG "result": [ 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG { 23:11:56 DEBUG { 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "result": [ 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "result": [ 23:11:56 DEBUG { 23:11:56 DEBUG { 23:11:56 DEBUG "result": [ 23:11:56 DEBUG "metric": { 23:11:56 DEBUG { 23:11:56 DEBUG "result": [ 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "status": "success", 23:11:56 DEBUG "pod": "ds-idrepo-2" 23:11:56 DEBUG { 23:11:56 DEBUG "result": [ 23:11:56 DEBUG { 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "metric": { 23:11:56 DEBUG { 23:11:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:11:56 DEBUG "metric": { 23:11:56 DEBUG { 23:11:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:11:56 DEBUG "data": { 23:11:56 DEBUG }, 23:11:56 DEBUG "metric": { 23:11:56 DEBUG { 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:11:56 DEBUG "instance": "10.142.0.107:9100" 23:11:56 DEBUG "metric": { 23:11:56 DEBUG }, 23:11:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:11:56 DEBUG "metric": { 23:11:56 DEBUG }, 23:11:56 DEBUG "resultType": "vector", 23:11:56 DEBUG "value": [ 23:11:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:11:56 DEBUG "metric": { 23:11:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:11:56 DEBUG }, 23:11:56 DEBUG }, 23:11:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:11:56 DEBUG "value": [ 23:11:56 DEBUG }, 23:11:56 DEBUG "pod": "ds-idrepo-2" 23:11:56 DEBUG "value": [ 23:11:56 DEBUG "result": [ 23:11:56 DEBUG 23:11:56 DEBUG }, 23:11:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG "value": [ 23:11:56 DEBUG }, 23:11:56 DEBUG 23:11:56 DEBUG "value": [ 23:11:56 DEBUG }, 23:11:56 DEBUG 23:11:56 DEBUG { 23:11:56 DEBUG 23:11:56 DEBUG "value": [ 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG "metric": { 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "instance": "10.142.0.107:9100" 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG }, 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 INFO 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 INFO [loop_until]: kubectl --namespace=xlou top node 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "pod": "ds-idrepo-0" 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:11:56 DEBUG 23:11:56 DEBUG [...] 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG 23:11:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:11:56 DEBUG "instance": "10.142.0.43:9100" 23:11:56 DEBUG 23:11:56 DEBUG }, 23:11:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:11:56 DEBUG 23:11:56 DEBUG }, 23:11:56 DEBUG }, 23:11:56 DEBUG 23:11:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:11:56 DEBUG 23:11:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:11:56 DEBUG }, 23:11:56 DEBUG }, 23:11:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:11:56 DEBUG "value": [ 23:11:56 DEBUG }, 23:11:56 DEBUG "pod": "ds-idrepo-0" 23:11:56 DEBUG "value": [ 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 23:11:56 DEBUG }, 23:11:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG "value": [ 23:11:56 DEBUG }, 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG }, 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG 23:11:56 DEBUG "value": [ 23:11:56 DEBUG }, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG "0.08" 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG "0" 23:11:56 DEBUG "0" 23:11:56 DEBUG "instance": "10.142.0.43:9100" 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "value": [ 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "54152.25435602659" 23:11:56 DEBUG "0.00033333333333333365" 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG ] 23:11:56 DEBUG "0" 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG ] 23:11:56 DEBUG ] 23:11:56 DEBUG }, 23:11:56 DEBUG "2298.1" 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG "2297.0666666666666" 23:11:56 DEBUG ] 23:11:56 DEBUG ] 23:11:56 DEBUG "6.711176502205273" 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG "0" 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG "value": [ 23:11:56 DEBUG ] 23:11:56 DEBUG "271836" 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG 1713219116, 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG "13898550272" 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG ] 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 DEBUG } 23:11:56 INFO [loop_until]: OK (rc = 0) 23:11:56 DEBUG --- stdout --- 23:11:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5078Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4505m 15% 10920Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10261m 34% 10642Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17570m 58% 3054Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1811m 6% 5566Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 6823m 22% 12855Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2982Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 7976m 26% 9112Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1736m 5% 4954Mi 4% gke-xlou-cdm-ds-0f601751-j488 4491m 15% 10404Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 73m 0% 2442Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 13317m 44% 10542Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12095m 40% 7149Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6838m 43% 5849Mi 9% 23:11:56 DEBUG --- stderr --- 23:11:56 DEBUG 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "instance": "10.142.0.107:9100" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "instance": "10.142.0.43:9100" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "0.00030000000000001136" 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219176" 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "0" 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "instance": "10.142.0.107:9100" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "instance": "10.142.0.43:9100" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "3238.5333333333333" 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG { 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:56 DEBUG "data": { 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG }, 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG "value": [ 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "instance": "10.142.0.107:9100" 23:12:56 DEBUG 23:12:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:12:56 DEBUG }, 23:12:56 DEBUG 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG [...] 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 ERROR [http_cmd]: ERROR 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG [...] 23:12:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:12:56 DEBUG [...] 23:12:56 ERROR --- status code --- 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG 23:12:56 DEBUG }, 23:12:56 DEBUG 23:12:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 ERROR --- http response --- 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG 23:12:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:12:56 DEBUG "instance": "10.142.0.43:9100" 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:56 DEBUG "0" 23:12:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:12:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:12:56 DEBUG }, 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG ] 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG { 23:12:56 DEBUG } 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG ] 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "12382.366666666665" 23:12:56 DEBUG "data": { 23:12:56 DEBUG } 23:12:56 DEBUG "4.655599219827763" 23:12:56 DEBUG ] 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG "result": [ 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG { 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG "metric": { 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:12:56 DEBUG } 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "2285.7666666666664" 23:12:56 DEBUG ] 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "pod": "ds-idrepo-2" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "pod": "ds-idrepo-0" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "0" 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "pod": "ds-idrepo-2" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "pod": "ds-idrepo-0" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "0" 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219176" 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "2159849472" 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "pod": "ds-idrepo-2" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "pod": "ds-idrepo-0" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "0" 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 INFO 23:12:56 INFO [loop_until]: kubectl --namespace=xlou top pods 23:12:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "pod": "ds-idrepo-2" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "pod": "ds-idrepo-0" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "0" 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "2285" 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "pod": "ds-idrepo-2" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "pod": "ds-idrepo-0" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "0" 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "pod": "ds-idrepo-2" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "pod": "ds-cts-2" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "0" 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [loop_until]: OK (rc = 0) 23:12:56 DEBUG --- stdout --- 23:12:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 10114m 8590Mi am-7d597cf7d6-8vr8f 7755m 7627Mi am-7d597cf7d6-9wz9h 12321m 8925Mi ds-cts-0 6431m 8522Mi ds-cts-1 4569m 8629Mi ds-cts-2 4536m 8751Mi ds-idrepo-0 12058m 4977Mi ds-idrepo-1 1674m 2693Mi ds-idrepo-2 1652m 2699Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 3m 1224Mi lodemon-56b95674f4-jwfm2 10m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6639m 1969Mi 23:12:56 DEBUG --- stderr --- 23:12:56 DEBUG 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "instance": "10.142.0.107:9100" 23:12:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:12:56 DEBUG }, 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "instance": "10.142.0.43:9100" 23:12:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:12:56 DEBUG }, 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "13899152384" 23:12:56 DEBUG "0" 23:12:56 DEBUG ] 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219176" 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG --- http response --- 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:56 DEBUG { 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 INFO [http_cmd]: http status code OK 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG --- status code --- 23:12:56 DEBUG { 23:12:56 DEBUG "data": { 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG { 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG --- http response --- 23:12:56 DEBUG "data": { 23:12:56 DEBUG "result": [ 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "data": { 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:56 DEBUG { 23:12:56 DEBUG "data": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG "result": [ 23:12:56 DEBUG ----- output ----- 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG "result": [ 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG { 23:12:56 DEBUG { 23:12:56 DEBUG { 23:12:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:12:56 DEBUG "result": [ 23:12:56 DEBUG { 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "status": "success", 23:12:56 DEBUG }, 23:12:56 DEBUG { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:12:56 DEBUG "data": { 23:12:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:12:56 DEBUG "data": { 23:12:56 DEBUG "value": [ 23:12:56 DEBUG "metric": { 23:12:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:12:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:12:56 DEBUG }, 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG }, 23:12:56 DEBUG "resultType": "vector", 23:12:56 DEBUG 23:12:56 DEBUG "instance": "10.142.0.107:9100" 23:12:56 DEBUG }, 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG "result": [ 23:12:56 DEBUG "value": [ 23:12:56 DEBUG "result": [ 23:12:56 DEBUG 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG { 23:12:56 DEBUG 23:12:56 DEBUG { 23:12:56 DEBUG 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "metric": { 23:12:56 DEBUG 23:12:56 DEBUG "metric": { 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:12:56 DEBUG 23:12:56 DEBUG "pod": "ds-idrepo-2" 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG }, 23:12:56 DEBUG [...] 23:12:56 DEBUG }, 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG [...] 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG }, 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG 23:12:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:12:56 DEBUG 23:12:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:12:56 DEBUG 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:12:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:12:56 DEBUG }, 23:12:56 DEBUG [...] 23:12:56 DEBUG }, 23:12:56 DEBUG [...] 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "instance": "10.142.0.43:9100" 23:12:56 DEBUG }, 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 23:12:56 DEBUG "16790.48316026592" 23:12:56 DEBUG }, 23:12:56 DEBUG "value": [ 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG 23:12:56 DEBUG ] 23:12:56 DEBUG "value": [ 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "992.6333333333333" 23:12:56 DEBUG 23:12:56 DEBUG "0.08" 23:12:56 DEBUG 23:12:56 DEBUG } 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG "2281.233333333333" 23:12:56 DEBUG "6.72971269829227" 23:12:56 DEBUG ] 23:12:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:12:56 DEBUG ] 23:12:56 DEBUG "pod": "ds-idrepo-0" 23:12:56 DEBUG ] 23:12:56 DEBUG "0" 23:12:56 DEBUG ] 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG }, 23:12:56 DEBUG } 23:12:56 DEBUG }, 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG "value": [ 23:12:56 DEBUG ] 23:12:56 DEBUG "value": [ 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG } 23:12:56 DEBUG 1713219176, 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG "394589" 23:12:56 DEBUG } 23:12:56 DEBUG "0" 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG ] 23:12:56 DEBUG ] 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 DEBUG } 23:12:56 INFO 23:12:56 INFO [loop_until]: kubectl --namespace=xlou top node 23:12:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:12:56 INFO [loop_until]: OK (rc = 0) 23:12:56 DEBUG --- stdout --- 23:12:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5069Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4665m 15% 10945Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10328m 34% 10647Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17515m 58% 3059Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1811m 6% 5567Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 6944m 23% 12891Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 2979Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 7970m 26% 9137Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1709m 5% 4946Mi 4% gke-xlou-cdm-ds-0f601751-j488 4635m 15% 10437Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 70m 0% 2448Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 12867m 43% 10545Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 11994m 40% 7153Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6882m 43% 5852Mi 9% 23:12:56 DEBUG --- stderr --- 23:12:56 DEBUG 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "instance": "10.142.0.107:9100" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "instance": "10.142.0.43:9100" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "0.00033333333333350387" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219236" 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "instance": "10.142.0.107:9100" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "instance": "10.142.0.43:9100" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "12458.966666666665" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "0" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 ERROR [http_cmd]: ERROR 23:13:56 ERROR --- status code --- 23:13:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:13:56 ERROR --- http response --- 23:13:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:13:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "pod": "ds-idrepo-2" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "ds-idrepo-0" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "0" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219236" 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "instance": "10.142.0.107:9100" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "instance": "10.142.0.43:9100" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "4583.766666666666" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "pod": "ds-idrepo-2" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "ds-idrepo-0" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "0" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "0" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "2403143680" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:13:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:13:56 DEBUG }, 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:13:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:13:56 DEBUG }, 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "3.184243224310904" 23:13:56 DEBUG "2276.0333333333333" 23:13:56 DEBUG ] 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "pod": "ds-idrepo-2" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "ds-idrepo-0" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "0" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "pod": "ds-idrepo-2" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "ds-cts-2" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "0" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "pod": "ds-idrepo-2" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "ds-idrepo-0" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "0" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219236" 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "2276.5666666666666" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "pod": "ds-idrepo-2" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "ds-idrepo-0" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "0" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "instance": "10.142.0.107:9100" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "instance": "10.142.0.43:9100" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "13899795456" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO 23:13:56 INFO [loop_until]: kubectl --namespace=xlou top pods 23:13:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "0" 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219236" 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 INFO [loop_until]: OK (rc = 0) 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- stdout --- 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12619m 8610Mi am-7d597cf7d6-8vr8f 10294m 7635Mi am-7d597cf7d6-9wz9h 15921m 8989Mi ds-cts-0 14315m 8529Mi ds-cts-1 10524m 8619Mi ds-cts-2 10241m 8757Mi ds-idrepo-0 13659m 4977Mi ds-idrepo-1 1788m 2734Mi ds-idrepo-2 1826m 2700Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1026Mi idm-b85f9d978-rl76f 3m 1224Mi lodemon-56b95674f4-jwfm2 7m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6742m 2199Mi 23:13:56 DEBUG --- status code --- 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG --- stderr --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:56 DEBUG 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG { 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:56 INFO [http_cmd]: http status code OK 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG { 23:13:56 DEBUG "data": { 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG --- status code --- 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:56 DEBUG "data": { 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG "data": { 23:13:56 DEBUG "result": [ 23:13:56 DEBUG { 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG --- http response --- 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG "result": [ 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG { 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "data": { 23:13:56 DEBUG "result": [ 23:13:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:56 DEBUG { 23:13:56 DEBUG { 23:13:56 DEBUG "result": [ 23:13:56 DEBUG "metric": { 23:13:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG { 23:13:56 DEBUG ----- output ----- 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "metric": { 23:13:56 DEBUG { 23:13:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:13:56 DEBUG }, 23:13:56 DEBUG "result": [ 23:13:56 DEBUG "metric": { 23:13:56 DEBUG { 23:13:56 DEBUG "data": { 23:13:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:13:56 DEBUG "metric": { 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG { 23:13:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:13:56 DEBUG "status": "success", 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG }, 23:13:56 DEBUG "pod": "ds-idrepo-2" 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG "metric": { 23:13:56 DEBUG }, 23:13:56 DEBUG "data": { 23:13:56 DEBUG "result": [ 23:13:56 DEBUG "value": [ 23:13:56 DEBUG }, 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:13:56 DEBUG "value": [ 23:13:56 DEBUG "resultType": "vector", 23:13:56 DEBUG { 23:13:56 DEBUG 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG }, 23:13:56 DEBUG 23:13:56 DEBUG "result": [ 23:13:56 DEBUG "metric": { 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG { 23:13:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "metric": { 23:13:56 DEBUG }, 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG "instance": "10.142.0.107:9100" 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG }, 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:13:56 DEBUG }, 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:13:56 DEBUG 23:13:56 DEBUG }, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 23:13:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:13:56 DEBUG 23:13:56 DEBUG [...] 23:13:56 DEBUG }, 23:13:56 DEBUG "pod": "ds-idrepo-0" 23:13:56 DEBUG "value": [ 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG 23:13:56 DEBUG }, 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "value": [ 23:13:56 DEBUG }, 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "0.08" 23:13:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:13:56 DEBUG "value": [ 23:13:56 DEBUG [...] 23:13:56 DEBUG 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "value": [ 23:13:56 DEBUG "7169600.4706785185" 23:13:56 DEBUG ] 23:13:56 DEBUG }, 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG 23:13:56 DEBUG 23:13:56 DEBUG "2276.1666666666665" 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG "value": [ 23:13:56 DEBUG "2303.4" 23:13:56 DEBUG 23:13:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:13:56 DEBUG ] 23:13:56 DEBUG "0" 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG ] 23:13:56 DEBUG 23:13:56 DEBUG }, 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG "6.808729608466398" 23:13:56 DEBUG } 23:13:56 DEBUG "instance": "10.142.0.43:9100" 23:13:56 DEBUG "value": [ 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG ] 23:13:56 DEBUG }, 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG "value": [ 23:13:56 DEBUG "413173" 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG 1713219236, 23:13:56 DEBUG "0" 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG ] 23:13:56 DEBUG ] 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 DEBUG } 23:13:56 INFO 23:13:56 INFO [loop_until]: kubectl --namespace=xlou top node 23:13:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:13:56 INFO [loop_until]: OK (rc = 0) 23:13:56 DEBUG --- stdout --- 23:13:56 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5092Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10555m 35% 10885Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13141m 44% 10672Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17086m 57% 3063Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1984m 6% 5569Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 15777m 52% 12838Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2974Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10078m 33% 9133Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1924m 6% 4987Mi 4% gke-xlou-cdm-ds-0f601751-j488 10386m 34% 10387Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 74m 0% 2451Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17065m 57% 10622Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13799m 46% 7158Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6758m 42% 6079Mi 10% 23:13:56 DEBUG --- stderr --- 23:13:56 DEBUG 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219296" 23:14:56 DEBUG --- status code --- 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG --- http response --- 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:56 DEBUG { 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG { 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG { 23:14:56 DEBUG "data": { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "data": { 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:14:56 DEBUG "data": { 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG "result": [ 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG { 23:14:56 DEBUG { 23:14:56 DEBUG "result": [ 23:14:56 DEBUG { 23:14:56 DEBUG "result": [ 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "metric": { 23:14:56 DEBUG { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG { 23:14:56 DEBUG "data": { 23:14:56 DEBUG "instance": "10.142.0.107:9100" 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "data": { 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "resultType": "vector", 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG }, 23:14:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "pod": "ds-idrepo-2" 23:14:56 DEBUG "result": [ 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG "value": [ 23:14:56 DEBUG }, 23:14:56 DEBUG "result": [ 23:14:56 DEBUG }, 23:14:56 DEBUG { 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG 23:14:56 DEBUG "value": [ 23:14:56 DEBUG { 23:14:56 DEBUG "value": [ 23:14:56 DEBUG "metric": { 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "metric": { 23:14:56 DEBUG 23:14:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:14:56 DEBUG 23:14:56 DEBUG "pod": "ds-idrepo-2" 23:14:56 DEBUG 23:14:56 DEBUG }, 23:14:56 DEBUG [...] 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG 23:14:56 DEBUG }, 23:14:56 DEBUG 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG { 23:14:56 DEBUG [...] 23:14:56 DEBUG "value": [ 23:14:56 DEBUG [...] 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 DEBUG 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "data": { 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "instance": "10.142.0.43:9100" 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG "data": { 23:14:56 ERROR [http_cmd]: ERROR 23:14:56 DEBUG "resultType": "vector", 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG 23:14:56 DEBUG [...] 23:14:56 DEBUG }, 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG "resultType": "vector", 23:14:56 ERROR --- status code --- 23:14:56 DEBUG "result": [ 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:14:56 DEBUG [...] 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG "pod": "ds-idrepo-0" 23:14:56 DEBUG 23:14:56 DEBUG "value": [ 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG "result": [ 23:14:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:14:56 DEBUG { 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG }, 23:14:56 DEBUG 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG }, 23:14:56 DEBUG 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG { 23:14:56 ERROR --- http response --- 23:14:56 DEBUG "metric": { 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG "0.00040000000000001514" 23:14:56 DEBUG "metric": { 23:14:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:14:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:14:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:56 DEBUG 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:14:56 DEBUG ] 23:14:56 DEBUG "instance": "10.142.0.107:9100" 23:14:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:14:56 DEBUG }, 23:14:56 DEBUG "2404016128" 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG "pod": "ds-idrepo-0" 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG "0" 23:14:56 DEBUG }, 23:14:56 DEBUG } 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG ] 23:14:56 DEBUG { 23:14:56 DEBUG }, 23:14:56 DEBUG { 23:14:56 DEBUG ] 23:14:56 DEBUG "value": [ 23:14:56 DEBUG ] 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG } 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "value": [ 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG } 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG } 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG ] 23:14:56 DEBUG "data": { 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "data": { 23:14:56 DEBUG ] 23:14:56 DEBUG "0" 23:14:56 DEBUG } 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG } 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "0" 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG } 23:14:56 DEBUG ] 23:14:56 DEBUG 23:14:56 DEBUG [...] 23:14:56 DEBUG } 23:14:56 DEBUG "result": [ 23:14:56 DEBUG ] 23:14:56 DEBUG "result": [ 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 DEBUG [...] 23:14:56 DEBUG 23:14:56 DEBUG { 23:14:56 DEBUG } 23:14:56 DEBUG { 23:14:56 DEBUG ] 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "metric": { 23:14:56 DEBUG ] 23:14:56 DEBUG "metric": { 23:14:56 DEBUG } 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:14:56 DEBUG } 23:14:56 DEBUG "instance": "10.142.0.107:9100" 23:14:56 DEBUG } 23:14:56 DEBUG 23:14:56 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:14:56 DEBUG }, 23:14:56 DEBUG } 23:14:56 DEBUG }, 23:14:56 DEBUG "instance": "10.142.0.43:9100" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG "value": [ 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "1176.0225599437977" 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG "12258.43333333333" 23:14:56 DEBUG ] 23:14:56 DEBUG [...] 23:14:56 DEBUG [...] 23:14:56 DEBUG ] 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:56 DEBUG } 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG } 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG ] 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG ] 23:14:56 DEBUG { 23:14:56 DEBUG } 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG } 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG } 23:14:56 DEBUG "pod": "ds-idrepo-2" 23:14:56 DEBUG "instance": "10.142.0.43:9100" 23:14:56 DEBUG } 23:14:56 DEBUG "data": { 23:14:56 DEBUG }, 23:14:56 DEBUG }, 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "value": [ 23:14:56 DEBUG "value": [ 23:14:56 DEBUG "result": [ 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG { 23:14:56 DEBUG "0" 23:14:56 DEBUG "3064.5333333333333" 23:14:56 DEBUG "metric": { 23:14:56 DEBUG ] 23:14:56 DEBUG ] 23:14:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 DEBUG }, 23:14:56 DEBUG ] 23:14:56 DEBUG ] 23:14:56 DEBUG "value": [ 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 DEBUG 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG [...] 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "2293.866666666667" 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "data": { 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "result": [ 23:14:56 DEBUG { 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "pod": "ds-idrepo-2" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG [...] 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "pod": "ds-idrepo-0" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "0" 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "data": { 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "result": [ 23:14:56 DEBUG { 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "pod": "ds-idrepo-2" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG [...] 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "pod": "ds-cts-2" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "0" 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "data": { 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "result": [ 23:14:56 DEBUG { 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "pod": "ds-idrepo-2" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG [...] 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "pod": "ds-idrepo-0" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "0" 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219296" 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG { 23:14:56 DEBUG { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "data": { 23:14:56 DEBUG "data": { 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "result": [ 23:14:56 DEBUG "result": [ 23:14:56 DEBUG { 23:14:56 DEBUG { 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:14:56 DEBUG "pod": "ds-idrepo-2" 23:14:56 DEBUG }, 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG [...] 23:14:56 DEBUG [...] 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:14:56 DEBUG "pod": "ds-idrepo-0" 23:14:56 DEBUG }, 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "2293.2" 23:14:56 DEBUG "0" 23:14:56 DEBUG ] 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 DEBUG ] 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "data": { 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "result": [ 23:14:56 DEBUG { 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "instance": "10.142.0.107:9100" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG [...] 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "instance": "10.142.0.43:9100" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "13900311552" 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219296" 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "data": { 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "result": [ 23:14:56 DEBUG { 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG [...] 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "0" 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "data": { 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "result": [ 23:14:56 DEBUG { 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "instance": "10.142.0.107:9100" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG [...] 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "instance": "10.142.0.43:9100" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "0" 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "data": { 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "result": [ 23:14:56 DEBUG { 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG [...] 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "pod": "ds-idrepo-2" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "0" 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 INFO 23:14:56 INFO [loop_until]: kubectl --namespace=xlou top pods 23:14:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219296" 23:14:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219296" 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "data": { 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "result": [ 23:14:56 DEBUG { 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG [...] 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "0.08" 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG ----- output ----- 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG { 23:14:56 DEBUG { 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "data": { 23:14:56 DEBUG "data": { 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG "data": { 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:14:56 DEBUG "result": [ 23:14:56 DEBUG "result": [ 23:14:56 DEBUG "result": [ 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG { 23:14:56 INFO [http_cmd]: http status code OK 23:14:56 DEBUG { 23:14:56 DEBUG { 23:14:56 DEBUG { 23:14:56 DEBUG "metric": { 23:14:56 DEBUG --- status code --- 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:14:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:14:56 DEBUG "pod": "ds-idrepo-2" 23:14:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:14:56 DEBUG "data": { 23:14:56 DEBUG }, 23:14:56 DEBUG --- http response --- 23:14:56 DEBUG }, 23:14:56 DEBUG }, 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG "value": [ 23:14:56 DEBUG "value": [ 23:14:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:14:56 DEBUG "value": [ 23:14:56 DEBUG "result": [ 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG ----- output ----- 23:14:56 DEBUG 23:14:56 DEBUG { 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG { 23:14:56 DEBUG 23:14:56 DEBUG "metric": { 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "status": "success", 23:14:56 DEBUG 23:14:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:14:56 DEBUG [...] 23:14:56 DEBUG [...] 23:14:56 DEBUG "data": { 23:14:56 DEBUG [...] 23:14:56 DEBUG }, 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "resultType": "vector", 23:14:56 DEBUG 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "result": [ 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG { 23:14:56 DEBUG 23:14:56 DEBUG 23:14:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:14:56 DEBUG "pod": "ds-idrepo-0" 23:14:56 DEBUG "metric": { 23:14:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:14:56 DEBUG 23:14:56 DEBUG }, 23:14:56 DEBUG }, 23:14:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:14:56 DEBUG }, 23:14:56 DEBUG [...] 23:14:56 DEBUG "value": [ 23:14:56 DEBUG "value": [ 23:14:56 DEBUG }, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG "value": [ 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG 23:14:56 DEBUG "2296.7" 23:14:56 DEBUG "0" 23:14:56 DEBUG 23:14:56 DEBUG "2284.866666666667" 23:14:56 DEBUG 23:14:56 DEBUG ] 23:14:56 DEBUG ] 23:14:56 DEBUG 23:14:56 DEBUG ] 23:14:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 DEBUG 23:14:56 DEBUG } 23:14:56 DEBUG }, 23:14:56 DEBUG ] 23:14:56 DEBUG ] 23:14:56 DEBUG [...] 23:14:56 DEBUG ] 23:14:56 DEBUG "value": [ 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 DEBUG 23:14:56 DEBUG } 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 DEBUG 23:14:56 DEBUG } 23:14:56 DEBUG "411823" 23:14:56 DEBUG 23:14:56 DEBUG ] 23:14:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:14:56 DEBUG } 23:14:56 DEBUG }, 23:14:56 DEBUG ] 23:14:56 DEBUG "value": [ 23:14:56 DEBUG } 23:14:56 DEBUG 1713219296, 23:14:56 DEBUG } 23:14:56 DEBUG "6.709719221283859" 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG ] 23:14:56 DEBUG } 23:14:56 DEBUG } 23:14:56 INFO [loop_until]: OK (rc = 0) 23:14:56 DEBUG --- stdout --- 23:14:56 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12413m 8622Mi am-7d597cf7d6-8vr8f 10068m 9037Mi am-7d597cf7d6-9wz9h 16121m 9003Mi ds-cts-0 14515m 8518Mi ds-cts-1 10736m 8614Mi ds-cts-2 10443m 8737Mi ds-idrepo-0 13425m 4977Mi ds-idrepo-1 1833m 2735Mi ds-idrepo-2 1832m 2757Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1026Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 10m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6570m 2201Mi 23:14:56 DEBUG --- stderr --- 23:14:56 DEBUG 23:14:56 INFO 23:14:56 INFO [loop_until]: kubectl --namespace=xlou top node 23:14:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:14:57 INFO [loop_until]: OK (rc = 0) 23:14:57 DEBUG --- stdout --- 23:14:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5077Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10694m 35% 10884Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13097m 43% 10679Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17531m 58% 3063Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1963m 6% 5621Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 15895m 53% 12832Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2964Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10169m 34% 10545Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1909m 6% 4985Mi 4% gke-xlou-cdm-ds-0f601751-j488 10420m 34% 10384Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 2462Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17078m 57% 10648Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13277m 44% 7153Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6851m 43% 6082Mi 10% 23:14:57 DEBUG --- stderr --- 23:14:57 DEBUG 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:15:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "0.004843925492002796" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219356" 23:15:56 ERROR [http_cmd]: ERROR 23:15:56 ERROR --- status code --- 23:15:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:15:56 ERROR --- http response --- 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219356" 23:15:56 DEBUG --- status code --- 23:15:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG --- http response --- 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG { 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG { 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG { 23:15:56 DEBUG { 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "data": { 23:15:56 DEBUG "data": { 23:15:56 DEBUG "data": { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:15:56 DEBUG "resultType": "vector", 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "instance": "10.142.0.107:9100" 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG }, 23:15:56 DEBUG "result": [ 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG "result": [ 23:15:56 DEBUG "result": [ 23:15:56 DEBUG }, 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG "value": [ 23:15:56 DEBUG { 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG { 23:15:56 DEBUG { 23:15:56 DEBUG "value": [ 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG 23:15:56 DEBUG "metric": { 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG 23:15:56 DEBUG --- http response --- 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG 23:15:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:15:56 DEBUG "pod": "ds-idrepo-2" 23:15:56 DEBUG 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG 23:15:56 DEBUG }, 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG }, 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG }, 23:15:56 DEBUG 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG [...] 23:15:56 DEBUG "value": [ 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG "value": [ 23:15:56 DEBUG { 23:15:56 DEBUG "value": [ 23:15:56 DEBUG [...] 23:15:56 DEBUG { 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:56 DEBUG "data": { 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "data": { 23:15:56 DEBUG 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:15:56 DEBUG { 23:15:56 DEBUG [...] 23:15:56 DEBUG [...] 23:15:56 DEBUG { 23:15:56 DEBUG "result": [ 23:15:56 DEBUG [...] 23:15:56 DEBUG "instance": "10.142.0.43:9100" 23:15:56 DEBUG "result": [ 23:15:56 DEBUG }, 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG { 23:15:56 DEBUG 23:15:56 DEBUG }, 23:15:56 DEBUG { 23:15:56 DEBUG "value": [ 23:15:56 DEBUG "data": { 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "data": { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG 23:15:56 DEBUG "value": [ 23:15:56 DEBUG "metric": { 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:15:56 DEBUG 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "instance": "10.142.0.107:9100" 23:15:56 DEBUG "2407206912" 23:15:56 DEBUG "result": [ 23:15:56 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:15:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:15:56 DEBUG "result": [ 23:15:56 DEBUG }, 23:15:56 DEBUG "pod": "ds-idrepo-0" 23:15:56 DEBUG "11961.933333333332" 23:15:56 DEBUG }, 23:15:56 DEBUG ] 23:15:56 DEBUG { 23:15:56 DEBUG }, 23:15:56 DEBUG }, 23:15:56 DEBUG { 23:15:56 DEBUG "value": [ 23:15:56 DEBUG }, 23:15:56 DEBUG ] 23:15:56 DEBUG "value": [ 23:15:56 DEBUG } 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "value": [ 23:15:56 DEBUG "value": [ 23:15:56 DEBUG "metric": { 23:15:56 DEBUG 23:15:56 DEBUG "value": [ 23:15:56 DEBUG } 23:15:56 DEBUG 23:15:56 DEBUG ] 23:15:56 DEBUG "pod": "ds-idrepo-2" 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "instance": "10.142.0.107:9100" 23:15:56 DEBUG 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG ] 23:15:56 DEBUG 23:15:56 DEBUG } 23:15:56 DEBUG }, 23:15:56 DEBUG "0" 23:15:56 DEBUG "2289.2666666666664" 23:15:56 DEBUG }, 23:15:56 DEBUG 23:15:56 DEBUG "0" 23:15:56 DEBUG } 23:15:56 DEBUG 23:15:56 DEBUG } 23:15:56 DEBUG "value": [ 23:15:56 DEBUG ] 23:15:56 DEBUG ] 23:15:56 DEBUG "value": [ 23:15:56 DEBUG [...] 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG } 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG ] 23:15:56 DEBUG ] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG ] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG } 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 DEBUG [...] 23:15:56 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:15:56 DEBUG } 23:15:56 DEBUG "instance": "10.142.0.43:9100" 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG }, 23:15:56 DEBUG }, 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "value": [ 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "pod": "ds-idrepo-0" 23:15:56 DEBUG "instance": "10.142.0.43:9100" 23:15:56 DEBUG "574.4839072883549" 23:15:56 DEBUG "3035.733333333333" 23:15:56 DEBUG }, 23:15:56 DEBUG }, 23:15:56 DEBUG ] 23:15:56 DEBUG ] 23:15:56 DEBUG "value": [ 23:15:56 DEBUG "value": [ 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG ] 23:15:56 DEBUG ] 23:15:56 DEBUG "0" 23:15:56 DEBUG "0.00043333333333350765" 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "pod": "ds-idrepo-2" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "pod": "ds-cts-2" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "0" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "pod": "ds-idrepo-2" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "pod": "ds-idrepo-0" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "0" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "pod": "ds-idrepo-2" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "pod": "ds-idrepo-0" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "0" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "pod": "ds-idrepo-2" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "pod": "ds-idrepo-0" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "0" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "2289.5333333333333" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "instance": "10.142.0.107:9100" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "instance": "10.142.0.43:9100" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "13900925952" 23:15:56 DEBUG ] 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:15:56 DEBUG } 23:15:56 DEBUG ----- output ----- 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG ] 23:15:56 DEBUG { 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG } 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG } 23:15:56 DEBUG "data": { 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "12968.602329970996" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "instance": "10.142.0.107:9100" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "instance": "10.142.0.43:9100" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "0.0006666666666666747" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "0" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "411604.5" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "0.0805" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219356" 23:15:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219356" 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "pod": "ds-idrepo-2" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "pod": "ds-idrepo-0" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "0" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "6.683761283717216" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "2284.1666666666665" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO [http_cmd]: http status code OK 23:15:56 DEBUG --- status code --- 23:15:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:15:56 DEBUG --- http response --- 23:15:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:15:56 DEBUG ----- output ----- 23:15:56 DEBUG { 23:15:56 DEBUG "status": "success", 23:15:56 DEBUG "data": { 23:15:56 DEBUG "resultType": "vector", 23:15:56 DEBUG "result": [ 23:15:56 DEBUG { 23:15:56 DEBUG "metric": { 23:15:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG [...] 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG 23:15:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:15:56 DEBUG }, 23:15:56 DEBUG "value": [ 23:15:56 DEBUG 1713219356, 23:15:56 DEBUG "2289.3" 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG ] 23:15:56 DEBUG } 23:15:56 DEBUG } 23:15:56 INFO 23:15:56 INFO [loop_until]: kubectl --namespace=xlou top pods 23:15:56 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:15:57 INFO [loop_until]: OK (rc = 0) 23:15:57 DEBUG --- stdout --- 23:15:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12231m 8628Mi am-7d597cf7d6-8vr8f 9843m 9072Mi am-7d597cf7d6-9wz9h 15925m 8985Mi ds-cts-0 15031m 8526Mi ds-cts-1 10876m 8595Mi ds-cts-2 10544m 8753Mi ds-idrepo-0 13309m 4977Mi ds-idrepo-1 1808m 2734Mi ds-idrepo-2 1800m 2757Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 10m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6698m 2203Mi 23:15:57 DEBUG --- stderr --- 23:15:57 DEBUG 23:15:57 INFO 23:15:57 INFO [loop_until]: kubectl --namespace=xlou top node 23:15:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:15:57 INFO [loop_until]: OK (rc = 0) 23:15:57 DEBUG --- stdout --- 23:15:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5086Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10883m 36% 10863Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12834m 42% 10672Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17611m 58% 3068Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2003m 6% 5633Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16315m 54% 12855Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2968Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10311m 34% 10578Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1932m 6% 4989Mi 4% gke-xlou-cdm-ds-0f601751-j488 10670m 35% 10401Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2455Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16817m 56% 10631Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13814m 46% 7150Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6884m 43% 6083Mi 10% 23:15:57 DEBUG --- stderr --- 23:15:57 DEBUG 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219416" 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "instance": "10.142.0.107:9100" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG "instance": "10.142.0.43:9100" 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "4452.566666666667" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "pod": "ds-idrepo-2" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "pod": "ds-idrepo-0" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "0" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "0" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 ERROR [http_cmd]: ERROR 23:16:56 ERROR --- status code --- 23:16:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:16:56 ERROR --- http response --- 23:16:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:16:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "6.703238900288175" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "2407096320" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219416" 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "instance": "10.142.0.107:9100" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "instance": "10.142.0.43:9100" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "12177.899999999998" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG { 23:16:56 DEBUG { 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "data": { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "pod": "ds-idrepo-2" 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "instance": "10.142.0.107:9100" 23:16:56 DEBUG }, 23:16:56 DEBUG "result": [ 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG { 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG "metric": { 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG }, 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG "value": [ 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "pod": "ds-idrepo-0" 23:16:56 DEBUG [...] 23:16:56 DEBUG "instance": "10.142.0.43:9100" 23:16:56 DEBUG }, 23:16:56 DEBUG 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "0" 23:16:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:16:56 DEBUG "0.00036666666666652263" 23:16:56 DEBUG ] 23:16:56 DEBUG }, 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG "value": [ 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG "0" 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG } 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG ] 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:56 DEBUG } 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG } 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "2289.1" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "pod": "ds-idrepo-2" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "pod": "ds-cts-2" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "0" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "pod": "ds-idrepo-2" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "pod": "ds-idrepo-0" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "0" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "pod": "ds-idrepo-2" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "pod": "ds-idrepo-0" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "0" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219416" 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "0" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "instance": "10.142.0.107:9100" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "instance": "10.142.0.43:9100" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "13901511680" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219416" 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG { 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:16:56 DEBUG "data": { 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG }, 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "value": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "pod": "ds-idrepo-2" 23:16:56 DEBUG 23:16:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:16:56 DEBUG }, 23:16:56 DEBUG 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG [...] 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG }, 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG 23:16:56 DEBUG "pod": "ds-idrepo-0" 23:16:56 DEBUG "411771.5" 23:16:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:16:56 DEBUG }, 23:16:56 DEBUG ] 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG } 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG ] 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "0" 23:16:56 DEBUG } 23:16:56 DEBUG "2288.0666666666666" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "6625.099108813549" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "instance": "10.142.0.107:9100" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "instance": "10.142.0.43:9100" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "0.0006666666666666672" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219416" 23:16:56 INFO [http_cmd]: http status code OK 23:16:56 DEBUG --- status code --- 23:16:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:56 DEBUG --- http response --- 23:16:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:56 DEBUG ----- output ----- 23:16:56 DEBUG { 23:16:56 DEBUG "status": "success", 23:16:56 DEBUG "data": { 23:16:56 DEBUG "resultType": "vector", 23:16:56 DEBUG "result": [ 23:16:56 DEBUG { 23:16:56 DEBUG "metric": { 23:16:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG [...] 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG 23:16:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:16:56 DEBUG }, 23:16:56 DEBUG "value": [ 23:16:56 DEBUG 1713219416, 23:16:56 DEBUG "0.081" 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG ] 23:16:56 DEBUG } 23:16:56 DEBUG } 23:16:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219417" 23:16:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219417" 23:16:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219417" 23:16:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219417" 23:16:57 INFO [http_cmd]: http status code OK 23:16:57 DEBUG --- status code --- 23:16:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:57 DEBUG --- http response --- 23:16:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:57 DEBUG ----- output ----- 23:16:57 DEBUG { 23:16:57 DEBUG "status": "success", 23:16:57 DEBUG "data": { 23:16:57 DEBUG "resultType": "vector", 23:16:57 DEBUG "result": [ 23:16:57 DEBUG { 23:16:57 DEBUG "metric": { 23:16:57 DEBUG "pod": "ds-idrepo-2" 23:16:57 DEBUG }, 23:16:57 DEBUG "value": [ 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG [...] 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG "pod": "ds-idrepo-0" 23:16:57 DEBUG }, 23:16:57 DEBUG "value": [ 23:16:57 DEBUG 1713219417, 23:16:57 DEBUG "0" 23:16:57 DEBUG ] 23:16:57 DEBUG } 23:16:57 DEBUG ] 23:16:57 DEBUG } 23:16:57 DEBUG } 23:16:57 INFO [http_cmd]: http status code OK 23:16:57 DEBUG --- status code --- 23:16:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:57 DEBUG --- http response --- 23:16:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:57 DEBUG ----- output ----- 23:16:57 DEBUG { 23:16:57 DEBUG "status": "success", 23:16:57 DEBUG "data": { 23:16:57 DEBUG "resultType": "vector", 23:16:57 DEBUG "result": [ 23:16:57 DEBUG { 23:16:57 DEBUG "metric": { 23:16:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:16:57 DEBUG }, 23:16:57 DEBUG "value": [ 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG [...] 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:16:57 DEBUG }, 23:16:57 DEBUG "value": [ 23:16:57 DEBUG 1713219417, 23:16:57 DEBUG "2275.5666666666666" 23:16:57 DEBUG ] 23:16:57 DEBUG } 23:16:57 DEBUG ] 23:16:57 DEBUG } 23:16:57 DEBUG } 23:16:57 INFO [http_cmd]: http status code OK 23:16:57 DEBUG --- status code --- 23:16:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:57 DEBUG --- http response --- 23:16:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:16:57 DEBUG ----- output ----- 23:16:57 DEBUG { 23:16:57 DEBUG "status": "success", 23:16:57 DEBUG "data": { 23:16:57 DEBUG "resultType": "vector", 23:16:57 DEBUG "result": [ 23:16:57 DEBUG { 23:16:57 DEBUG "metric": { 23:16:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:16:57 DEBUG }, 23:16:57 DEBUG "value": [ 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG [...] 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:16:57 DEBUG }, 23:16:57 DEBUG "value": [ 23:16:57 DEBUG 1713219417, 23:16:57 DEBUG "6.800813848891663" 23:16:57 DEBUG ] 23:16:57 DEBUG } 23:16:57 DEBUG ] 23:16:57 DEBUG } 23:16:57 DEBUG } 23:16:57 INFO [http_cmd]: http status code OK 23:16:57 DEBUG --- status code --- 23:16:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:16:57 DEBUG --- http response --- 23:16:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:16:57 DEBUG ----- output ----- 23:16:57 DEBUG { 23:16:57 DEBUG "status": "success", 23:16:57 DEBUG "data": { 23:16:57 DEBUG "resultType": "vector", 23:16:57 DEBUG "result": [ 23:16:57 DEBUG { 23:16:57 DEBUG "metric": { 23:16:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:16:57 DEBUG }, 23:16:57 DEBUG "value": [ 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG [...] 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG 23:16:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:16:57 DEBUG }, 23:16:57 DEBUG "value": [ 23:16:57 DEBUG 1713219417, 23:16:57 DEBUG "2299.5333333333333" 23:16:57 DEBUG ] 23:16:57 DEBUG } 23:16:57 DEBUG ] 23:16:57 DEBUG } 23:16:57 DEBUG } 23:16:57 INFO 23:16:57 INFO [loop_until]: kubectl --namespace=xlou top pods 23:16:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:16:57 INFO [loop_until]: OK (rc = 0) 23:16:57 DEBUG --- stdout --- 23:16:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12787m 8614Mi am-7d597cf7d6-8vr8f 10190m 9072Mi am-7d597cf7d6-9wz9h 15953m 8988Mi ds-cts-0 15209m 8524Mi ds-cts-1 10929m 8620Mi ds-cts-2 10922m 8750Mi ds-idrepo-0 13322m 4977Mi ds-idrepo-1 1816m 2734Mi ds-idrepo-2 1870m 2757Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 1m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6697m 2206Mi 23:16:57 DEBUG --- stderr --- 23:16:57 DEBUG 23:16:57 INFO 23:16:57 INFO [loop_until]: kubectl --namespace=xlou top node 23:16:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:16:57 INFO [loop_until]: OK (rc = 0) 23:16:57 DEBUG --- stdout --- 23:16:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5077Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10944m 36% 10879Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13111m 43% 10683Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17005m 56% 3067Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1976m 6% 5637Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16195m 54% 12876Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2966Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10282m 34% 10399Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1905m 6% 4995Mi 4% gke-xlou-cdm-ds-0f601751-j488 10746m 35% 10396Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2456Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17202m 57% 10633Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13749m 46% 7152Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6882m 43% 6081Mi 10% 23:16:57 DEBUG --- stderr --- 23:16:57 DEBUG 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219476" 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG --- http response --- 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG { 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG "data": { 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG "result": [ 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:17:56 DEBUG { 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG "metric": { 23:17:56 DEBUG { 23:17:56 DEBUG "pod": "ds-idrepo-2" 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG }, 23:17:56 DEBUG "data": { 23:17:56 DEBUG "value": [ 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG 23:17:56 DEBUG "result": [ 23:17:56 DEBUG 23:17:56 DEBUG { 23:17:56 DEBUG 23:17:56 DEBUG "metric": { 23:17:56 DEBUG [...] 23:17:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:17:56 DEBUG 23:17:56 DEBUG }, 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG 23:17:56 DEBUG "value": [ 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG "pod": "ds-idrepo-0" 23:17:56 DEBUG 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG }, 23:17:56 DEBUG 23:17:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:17:56 DEBUG "value": [ 23:17:56 DEBUG [...] 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG 23:17:56 DEBUG "0" 23:17:56 DEBUG 23:17:56 DEBUG ] 23:17:56 DEBUG 23:17:56 DEBUG } 23:17:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:17:56 DEBUG ] 23:17:56 DEBUG }, 23:17:56 DEBUG } 23:17:56 DEBUG "value": [ 23:17:56 DEBUG } 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "6.704413528794546" 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219476" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219476" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 ERROR [http_cmd]: ERROR 23:17:56 ERROR --- status code --- 23:17:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:17:56 ERROR --- http response --- 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:17:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219476" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219476" 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG { 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG { 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG { 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG "data": { 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG "data": { 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG "data": { 23:17:56 DEBUG "resultType": "vector", 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG "result": [ 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG "result": [ 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG { 23:17:56 DEBUG "result": [ 23:17:56 DEBUG { 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG "metric": { 23:17:56 DEBUG { 23:17:56 DEBUG "metric": { 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:17:56 DEBUG "metric": { 23:17:56 DEBUG "instance": "10.142.0.107:9100" 23:17:56 DEBUG }, 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:17:56 DEBUG "instance": "10.142.0.107:9100" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG { 23:17:56 DEBUG 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG [...] 23:17:56 DEBUG "data": { 23:17:56 DEBUG 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG [...] 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG "result": [ 23:17:56 DEBUG 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG [...] 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG { 23:17:56 DEBUG 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG "metric": { 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG }, 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:17:56 DEBUG "instance": "10.142.0.43:9100" 23:17:56 DEBUG 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG "value": [ 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG }, 23:17:56 DEBUG }, 23:17:56 DEBUG "instance": "10.142.0.43:9100" 23:17:56 DEBUG { 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:17:56 DEBUG "value": [ 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:56 DEBUG "value": [ 23:17:56 DEBUG }, 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG "2295.633333333333" 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG "data": { 23:17:56 DEBUG ] 23:17:56 DEBUG { 23:17:56 DEBUG 23:17:56 DEBUG { 23:17:56 DEBUG "12117.066666666666" 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG } 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG ] 23:17:56 DEBUG "3203.0666666666666" 23:17:56 DEBUG "result": [ 23:17:56 DEBUG ] 23:17:56 DEBUG "data": { 23:17:56 DEBUG [...] 23:17:56 DEBUG "data": { 23:17:56 DEBUG } 23:17:56 DEBUG ] 23:17:56 DEBUG { 23:17:56 DEBUG } 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG "metric": { 23:17:56 DEBUG } 23:17:56 DEBUG "result": [ 23:17:56 DEBUG 23:17:56 DEBUG "result": [ 23:17:56 DEBUG } 23:17:56 DEBUG ] 23:17:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:17:56 DEBUG { 23:17:56 DEBUG 23:17:56 DEBUG { 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 DEBUG }, 23:17:56 DEBUG "metric": { 23:17:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:17:56 DEBUG "metric": { 23:17:56 DEBUG } 23:17:56 DEBUG "value": [ 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG "pod": "ds-idrepo-2" 23:17:56 DEBUG }, 23:17:56 DEBUG "instance": "10.142.0.107:9100" 23:17:56 DEBUG 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG }, 23:17:56 DEBUG 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG 23:17:56 DEBUG "0" 23:17:56 DEBUG 23:17:56 DEBUG [...] 23:17:56 DEBUG 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:17:56 DEBUG ] 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG } 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG [...] 23:17:56 DEBUG { 23:17:56 DEBUG ] 23:17:56 DEBUG [...] 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG } 23:17:56 DEBUG 23:17:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:17:56 DEBUG 23:17:56 DEBUG "data": { 23:17:56 DEBUG } 23:17:56 DEBUG 23:17:56 DEBUG }, 23:17:56 DEBUG 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG 23:17:56 DEBUG "value": [ 23:17:56 DEBUG "pod": "ds-cts-2" 23:17:56 DEBUG "result": [ 23:17:56 DEBUG "instance": "10.142.0.43:9100" 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG }, 23:17:56 DEBUG { 23:17:56 DEBUG }, 23:17:56 DEBUG "2409345024" 23:17:56 DEBUG "value": [ 23:17:56 DEBUG "metric": { 23:17:56 DEBUG "value": [ 23:17:56 DEBUG ] 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "pod": "ds-idrepo-2" 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG } 23:17:56 DEBUG "0" 23:17:56 DEBUG }, 23:17:56 DEBUG "0.000500000000000019" 23:17:56 DEBUG ] 23:17:56 DEBUG ] 23:17:56 DEBUG "value": [ 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 DEBUG 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 DEBUG ] 23:17:56 DEBUG 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 DEBUG [...] 23:17:56 DEBUG } 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG "pod": "ds-idrepo-0" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "0" 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG { 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG "data": { 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG "result": [ 23:17:56 DEBUG { 23:17:56 DEBUG "metric": { 23:17:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG [...] 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "0" 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG { 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG "data": { 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG "result": [ 23:17:56 DEBUG { 23:17:56 DEBUG "metric": { 23:17:56 DEBUG "pod": "ds-idrepo-2" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG [...] 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG "pod": "ds-idrepo-0" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "0" 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219476" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG { 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG "data": { 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG "result": [ 23:17:56 DEBUG { 23:17:56 DEBUG "metric": { 23:17:56 DEBUG "instance": "10.142.0.107:9100" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG [...] 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG "instance": "10.142.0.43:9100" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "13902162944" 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG { 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG "data": { 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG "result": [ 23:17:56 DEBUG { 23:17:56 DEBUG "metric": { 23:17:56 DEBUG "pod": "ds-idrepo-2" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG [...] 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG "pod": "ds-idrepo-0" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "0" 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG { 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG "data": { 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG "result": [ 23:17:56 DEBUG { 23:17:56 DEBUG "metric": { 23:17:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG [...] 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "413143.5" 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG { 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG "data": { 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG "result": [ 23:17:56 DEBUG { 23:17:56 DEBUG "metric": { 23:17:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG [...] 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "0" 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG { 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG "data": { 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG "result": [ 23:17:56 DEBUG { 23:17:56 DEBUG "metric": { 23:17:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG [...] 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "2296.133333333333" 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 INFO [http_cmd]: http status code OK 23:17:56 DEBUG --- status code --- 23:17:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:56 DEBUG --- http response --- 23:17:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:56 DEBUG ----- output ----- 23:17:56 DEBUG { 23:17:56 DEBUG "status": "success", 23:17:56 DEBUG "data": { 23:17:56 DEBUG "resultType": "vector", 23:17:56 DEBUG "result": [ 23:17:56 DEBUG { 23:17:56 DEBUG "metric": { 23:17:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG [...] 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG 23:17:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:17:56 DEBUG }, 23:17:56 DEBUG "value": [ 23:17:56 DEBUG 1713219476, 23:17:56 DEBUG "0.082" 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG ] 23:17:56 DEBUG } 23:17:56 DEBUG } 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219476" 23:17:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219476" 23:17:56 INFO [http_cmd]: http status code OK 23:17:57 INFO [http_cmd]: http status code OK 23:17:57 DEBUG --- status code --- 23:17:57 DEBUG --- status code --- 23:17:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:57 DEBUG --- http response --- 23:17:57 DEBUG --- http response --- 23:17:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:17:57 DEBUG ----- output ----- 23:17:57 DEBUG { 23:17:57 DEBUG "status": "success", 23:17:57 DEBUG "data": { 23:17:57 DEBUG "resultType": "vector", 23:17:57 DEBUG "result": [ 23:17:57 DEBUG { 23:17:57 DEBUG "metric": { 23:17:57 DEBUG "pod": "ds-idrepo-2" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG [...] 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG "pod": "ds-idrepo-0" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 1713219476, 23:17:57 DEBUG "0" 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG } 23:17:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:17:57 DEBUG ----- output ----- 23:17:57 DEBUG { 23:17:57 DEBUG "status": "success", 23:17:57 DEBUG "data": { 23:17:57 DEBUG "resultType": "vector", 23:17:57 DEBUG "result": [ 23:17:57 DEBUG { 23:17:57 DEBUG "metric": { 23:17:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG [...] 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 1713219476, 23:17:57 DEBUG "36715.05231285787" 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG } 23:17:57 INFO [http_cmd]: http status code OK 23:17:57 DEBUG --- status code --- 23:17:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:57 DEBUG --- http response --- 23:17:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:57 DEBUG ----- output ----- 23:17:57 DEBUG { 23:17:57 DEBUG "status": "success", 23:17:57 DEBUG "data": { 23:17:57 DEBUG "resultType": "vector", 23:17:57 DEBUG "result": [ 23:17:57 DEBUG { 23:17:57 DEBUG "metric": { 23:17:57 DEBUG "instance": "10.142.0.107:9100" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG [...] 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG "instance": "10.142.0.43:9100" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 1713219476, 23:17:57 DEBUG "0" 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG } 23:17:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219477" 23:17:57 INFO [http_cmd]: http status code OK 23:17:57 DEBUG --- status code --- 23:17:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:57 DEBUG --- http response --- 23:17:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:17:57 DEBUG ----- output ----- 23:17:57 DEBUG { 23:17:57 DEBUG "status": "success", 23:17:57 DEBUG "data": { 23:17:57 DEBUG "resultType": "vector", 23:17:57 DEBUG "result": [ 23:17:57 DEBUG { 23:17:57 DEBUG "metric": { 23:17:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG [...] 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 1713219477, 23:17:57 DEBUG "6.7463298609896585" 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG } 23:17:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219477" 23:17:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219477" 23:17:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219477" 23:17:57 INFO [http_cmd]: http status code OK 23:17:57 DEBUG --- status code --- 23:17:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:57 DEBUG --- http response --- 23:17:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:57 DEBUG ----- output ----- 23:17:57 DEBUG { 23:17:57 DEBUG "status": "success", 23:17:57 DEBUG "data": { 23:17:57 DEBUG "resultType": "vector", 23:17:57 DEBUG "result": [ 23:17:57 DEBUG { 23:17:57 DEBUG "metric": { 23:17:57 DEBUG "pod": "ds-idrepo-2" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG [...] 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG "pod": "ds-idrepo-0" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 1713219477, 23:17:57 DEBUG "0" 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG } 23:17:57 INFO [http_cmd]: http status code OK 23:17:57 DEBUG --- status code --- 23:17:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:57 DEBUG --- http response --- 23:17:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:57 DEBUG ----- output ----- 23:17:57 DEBUG { 23:17:57 DEBUG "status": "success", 23:17:57 DEBUG "data": { 23:17:57 DEBUG "resultType": "vector", 23:17:57 DEBUG "result": [ 23:17:57 DEBUG { 23:17:57 DEBUG "metric": { 23:17:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG [...] 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 1713219477, 23:17:57 DEBUG "2292.8" 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG } 23:17:57 INFO [http_cmd]: http status code OK 23:17:57 DEBUG --- status code --- 23:17:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:17:57 DEBUG --- http response --- 23:17:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:17:57 DEBUG ----- output ----- 23:17:57 DEBUG { 23:17:57 DEBUG "status": "success", 23:17:57 DEBUG "data": { 23:17:57 DEBUG "resultType": "vector", 23:17:57 DEBUG "result": [ 23:17:57 DEBUG { 23:17:57 DEBUG "metric": { 23:17:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG [...] 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG 23:17:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:17:57 DEBUG }, 23:17:57 DEBUG "value": [ 23:17:57 DEBUG 1713219477, 23:17:57 DEBUG "2294.1666666666665" 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG ] 23:17:57 DEBUG } 23:17:57 DEBUG } 23:17:57 INFO 23:17:57 INFO [loop_until]: kubectl --namespace=xlou top pods 23:17:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:17:57 INFO [loop_until]: OK (rc = 0) 23:17:57 DEBUG --- stdout --- 23:17:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12252m 8616Mi am-7d597cf7d6-8vr8f 10298m 8893Mi am-7d597cf7d6-9wz9h 16368m 9003Mi ds-cts-0 15444m 8526Mi ds-cts-1 11244m 8601Mi ds-cts-2 10701m 8746Mi ds-idrepo-0 13643m 4977Mi ds-idrepo-1 1828m 2734Mi ds-idrepo-2 1809m 2757Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 1m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6765m 2208Mi 23:17:57 DEBUG --- stderr --- 23:17:57 DEBUG 23:17:57 INFO 23:17:57 INFO [loop_until]: kubectl --namespace=xlou top node 23:17:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:17:57 INFO [loop_until]: OK (rc = 0) 23:17:57 DEBUG --- stdout --- 23:17:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5084Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10876m 36% 10876Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13039m 43% 10677Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17568m 58% 3069Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1974m 6% 5628Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 15970m 53% 12858Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2957Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10281m 34% 10386Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1910m 6% 4990Mi 4% gke-xlou-cdm-ds-0f601751-j488 10641m 35% 10404Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 58m 0% 2449Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17248m 57% 10648Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13842m 46% 7157Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6638m 41% 6081Mi 10% 23:17:57 DEBUG --- stderr --- 23:17:57 DEBUG 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219536" 23:18:56 ERROR [http_cmd]: ERROR 23:18:56 ERROR --- status code --- 23:18:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:18:56 ERROR --- http response --- 23:18:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:18:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219536" 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219536" 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219536" 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219536" 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 DEBUG --- status code --- 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 DEBUG --- status code --- 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG --- status code --- 23:18:56 DEBUG --- status code --- 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:18:56 DEBUG { 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG { 23:18:56 DEBUG { 23:18:56 DEBUG "data": { 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG "data": { 23:18:56 DEBUG "data": { 23:18:56 DEBUG "result": [ 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG { 23:18:56 DEBUG "result": [ 23:18:56 DEBUG "result": [ 23:18:56 DEBUG "metric": { 23:18:56 DEBUG { 23:18:56 DEBUG { 23:18:56 DEBUG "pod": "ds-idrepo-2" 23:18:56 DEBUG "metric": { 23:18:56 DEBUG "metric": { 23:18:56 DEBUG }, 23:18:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:18:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:18:56 DEBUG "value": [ 23:18:56 DEBUG }, 23:18:56 DEBUG }, 23:18:56 DEBUG 23:18:56 DEBUG "value": [ 23:18:56 DEBUG "value": [ 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG [...] 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG [...] 23:18:56 DEBUG [...] 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG "pod": "ds-idrepo-0" 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG }, 23:18:56 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:18:56 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:18:56 DEBUG "value": [ 23:18:56 DEBUG }, 23:18:56 DEBUG }, 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG "value": [ 23:18:56 DEBUG "value": [ 23:18:56 DEBUG "0" 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG ] 23:18:56 DEBUG "0" 23:18:56 DEBUG "0" 23:18:56 DEBUG } 23:18:56 DEBUG ] 23:18:56 DEBUG ] 23:18:56 DEBUG ] 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG ] 23:18:56 DEBUG ] 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219536" 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219536" 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219536" 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219536" 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219536" 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219536" 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219536" 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219536" 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219536" 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 DEBUG --- status code --- 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG { 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG "data": { 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG "result": [ 23:18:56 DEBUG { 23:18:56 DEBUG "metric": { 23:18:56 DEBUG "instance": "10.142.0.107:9100" 23:18:56 DEBUG }, 23:18:56 DEBUG "value": [ 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG [...] 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG "instance": "10.142.0.43:9100" 23:18:56 DEBUG }, 23:18:56 DEBUG "value": [ 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG "12260" 23:18:56 DEBUG ] 23:18:56 DEBUG } 23:18:56 DEBUG ] 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 DEBUG --- status code --- 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG --- http response --- 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:18:56 DEBUG --- status code --- 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG --- status code --- 23:18:56 DEBUG { 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:56 DEBUG "data": { 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG { 23:18:56 DEBUG "result": [ 23:18:56 DEBUG { 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG { 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG "data": { 23:18:56 DEBUG "metric": { 23:18:56 DEBUG "data": { 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG "result": [ 23:18:56 DEBUG }, 23:18:56 DEBUG "result": [ 23:18:56 DEBUG { 23:18:56 DEBUG "value": [ 23:18:56 DEBUG { 23:18:56 DEBUG "metric": { 23:18:56 DEBUG 23:18:56 DEBUG "metric": { 23:18:56 DEBUG "instance": "10.142.0.107:9100" 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 DEBUG 23:18:56 DEBUG "instance": "10.142.0.107:9100" 23:18:56 DEBUG }, 23:18:56 DEBUG --- status code --- 23:18:56 DEBUG --- status code --- 23:18:56 DEBUG 23:18:56 DEBUG }, 23:18:56 DEBUG "value": [ 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG [...] 23:18:56 DEBUG "value": [ 23:18:56 DEBUG 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:18:56 DEBUG 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG --- status code --- 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG 23:18:56 DEBUG --- status code --- 23:18:56 DEBUG 23:18:56 DEBUG [...] 23:18:56 DEBUG { 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG { 23:18:56 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG [...] 23:18:56 DEBUG 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG }, 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG "data": { 23:18:56 DEBUG "data": { 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:56 DEBUG "value": [ 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:18:56 DEBUG "instance": "10.142.0.43:9100" 23:18:56 DEBUG "instance": "10.142.0.43:9100" 23:18:56 DEBUG --- status code --- 23:18:56 DEBUG "result": [ 23:18:56 DEBUG "result": [ 23:18:56 DEBUG { 23:18:56 DEBUG "744.9561793313226" 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG }, 23:18:56 DEBUG }, 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG { 23:18:56 DEBUG { 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG ] 23:18:56 DEBUG { 23:18:56 DEBUG "value": [ 23:18:56 DEBUG "value": [ 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG "metric": { 23:18:56 DEBUG "metric": { 23:18:56 DEBUG "data": { 23:18:56 DEBUG } 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:18:56 DEBUG "pod": "ds-idrepo-2" 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG ] 23:18:56 DEBUG "data": { 23:18:56 DEBUG "4324.5" 23:18:56 DEBUG "0.0004666666666665264" 23:18:56 DEBUG }, 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG }, 23:18:56 DEBUG "result": [ 23:18:56 DEBUG } 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG ] 23:18:56 DEBUG ] 23:18:56 DEBUG "value": [ 23:18:56 DEBUG { 23:18:56 DEBUG "value": [ 23:18:56 DEBUG { 23:18:56 DEBUG } 23:18:56 DEBUG "result": [ 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG 23:18:56 DEBUG "metric": { 23:18:56 DEBUG { 23:18:56 DEBUG ] 23:18:56 DEBUG ] 23:18:56 DEBUG 23:18:56 DEBUG "data": { 23:18:56 DEBUG 23:18:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:18:56 DEBUG "metric": { 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG 23:18:56 DEBUG }, 23:18:56 DEBUG "pod": "ds-idrepo-2" 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG [...] 23:18:56 DEBUG "result": [ 23:18:56 DEBUG [...] 23:18:56 DEBUG "value": [ 23:18:56 DEBUG }, 23:18:56 DEBUG 23:18:56 DEBUG { 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG "value": [ 23:18:56 DEBUG 23:18:56 DEBUG "metric": { 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG "pod": "ds-idrepo-2" 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:18:56 DEBUG }, 23:18:56 DEBUG "pod": "ds-idrepo-0" 23:18:56 DEBUG [...] 23:18:56 DEBUG 23:18:56 DEBUG }, 23:18:56 DEBUG "value": [ 23:18:56 DEBUG }, 23:18:56 DEBUG 23:18:56 DEBUG [...] 23:18:56 DEBUG "value": [ 23:18:56 DEBUG 23:18:56 DEBUG "value": [ 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG "2282.5666666666666" 23:18:56 DEBUG 23:18:56 DEBUG "0" 23:18:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:18:56 DEBUG 23:18:56 DEBUG ] 23:18:56 DEBUG [...] 23:18:56 DEBUG ] 23:18:56 DEBUG }, 23:18:56 DEBUG "pod": "ds-idrepo-0" 23:18:56 DEBUG } 23:18:56 DEBUG 23:18:56 DEBUG } 23:18:56 DEBUG "value": [ 23:18:56 DEBUG }, 23:18:56 DEBUG ] 23:18:56 DEBUG 23:18:56 DEBUG ] 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG "value": [ 23:18:56 DEBUG } 23:18:56 DEBUG 23:18:56 DEBUG } 23:18:56 DEBUG "2413146112" 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG } 23:18:56 DEBUG "pod": "ds-cts-2" 23:18:56 DEBUG } 23:18:56 DEBUG ] 23:18:56 DEBUG "0" 23:18:56 DEBUG }, 23:18:56 DEBUG } 23:18:56 DEBUG ] 23:18:56 DEBUG "value": [ 23:18:56 DEBUG ] 23:18:56 DEBUG } 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG } 23:18:56 DEBUG ] 23:18:56 DEBUG "0" 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG ] 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG ] 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219536" 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219536" 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219536" 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 DEBUG --- status code --- 23:18:56 INFO [http_cmd]: http status code OK 23:18:56 DEBUG --- status code --- 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG --- status code --- 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG --- http response --- 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG ----- output ----- 23:18:56 DEBUG { 23:18:56 DEBUG { 23:18:56 DEBUG { 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG "status": "success", 23:18:56 DEBUG "data": { 23:18:56 DEBUG "data": { 23:18:56 DEBUG "data": { 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG "resultType": "vector", 23:18:56 DEBUG "result": [ 23:18:56 DEBUG "result": [ 23:18:56 DEBUG "result": [ 23:18:56 DEBUG { 23:18:56 DEBUG { 23:18:56 DEBUG { 23:18:56 DEBUG "metric": { 23:18:56 DEBUG "metric": { 23:18:56 DEBUG "metric": { 23:18:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:18:56 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:18:56 DEBUG "instance": "10.142.0.107:9100" 23:18:56 DEBUG }, 23:18:56 DEBUG }, 23:18:56 DEBUG }, 23:18:56 DEBUG "value": [ 23:18:56 DEBUG "value": [ 23:18:56 DEBUG "value": [ 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG [...] 23:18:56 DEBUG [...] 23:18:56 DEBUG [...] 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG 23:18:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:18:56 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:18:56 DEBUG "instance": "10.142.0.43:9100" 23:18:56 DEBUG }, 23:18:56 DEBUG }, 23:18:56 DEBUG }, 23:18:56 DEBUG "value": [ 23:18:56 DEBUG "value": [ 23:18:56 DEBUG "value": [ 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG 1713219536, 23:18:56 DEBUG "2283.633333333333" 23:18:56 DEBUG "0" 23:18:56 DEBUG "13902922752" 23:18:56 DEBUG ] 23:18:56 DEBUG ] 23:18:56 DEBUG ] 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG ] 23:18:56 DEBUG ] 23:18:56 DEBUG ] 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 DEBUG } 23:18:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219536" 23:18:57 INFO [http_cmd]: http status code OK 23:18:57 DEBUG --- status code --- 23:18:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:57 DEBUG --- http response --- 23:18:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:57 DEBUG ----- output ----- 23:18:57 DEBUG { 23:18:57 DEBUG "status": "success", 23:18:57 DEBUG "data": { 23:18:57 DEBUG "resultType": "vector", 23:18:57 DEBUG "result": [ 23:18:57 DEBUG { 23:18:57 DEBUG "metric": { 23:18:57 DEBUG "pod": "ds-idrepo-2" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG [...] 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG "pod": "ds-idrepo-0" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 1713219536, 23:18:57 DEBUG "0" 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG } 23:18:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219537" 23:18:57 INFO [http_cmd]: http status code OK 23:18:57 DEBUG --- status code --- 23:18:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:57 DEBUG --- http response --- 23:18:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219537" 23:18:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:57 DEBUG ----- output ----- 23:18:57 DEBUG { 23:18:57 DEBUG "status": "success", 23:18:57 DEBUG "data": { 23:18:57 DEBUG "resultType": "vector", 23:18:57 DEBUG "result": [ 23:18:57 DEBUG { 23:18:57 DEBUG "metric": { 23:18:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG [...] 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 1713219537, 23:18:57 DEBUG "413641.5" 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG } 23:18:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219537" 23:18:57 INFO [http_cmd]: http status code OK 23:18:57 DEBUG --- status code --- 23:18:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:57 DEBUG --- http response --- 23:18:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:57 DEBUG ----- output ----- 23:18:57 DEBUG { 23:18:57 DEBUG "status": "success", 23:18:57 DEBUG "data": { 23:18:57 DEBUG "resultType": "vector", 23:18:57 DEBUG "result": [ 23:18:57 DEBUG { 23:18:57 DEBUG "metric": { 23:18:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG [...] 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 1713219537, 23:18:57 DEBUG "0.082" 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG } 23:18:57 INFO [http_cmd]: http status code OK 23:18:57 DEBUG --- status code --- 23:18:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:57 DEBUG --- http response --- 23:18:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:57 DEBUG ----- output ----- 23:18:57 DEBUG { 23:18:57 DEBUG "status": "success", 23:18:57 DEBUG "data": { 23:18:57 DEBUG "resultType": "vector", 23:18:57 DEBUG "result": [ 23:18:57 DEBUG { 23:18:57 DEBUG "metric": { 23:18:57 DEBUG "instance": "10.142.0.107:9100" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG [...] 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG "instance": "10.142.0.43:9100" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 1713219537, 23:18:57 DEBUG "0" 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG } 23:18:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219537" 23:18:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219537" 23:18:57 INFO [http_cmd]: http status code OK 23:18:57 DEBUG --- status code --- 23:18:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:57 DEBUG --- http response --- 23:18:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:18:57 DEBUG ----- output ----- 23:18:57 DEBUG { 23:18:57 DEBUG "status": "success", 23:18:57 DEBUG "data": { 23:18:57 DEBUG "resultType": "vector", 23:18:57 DEBUG "result": [ 23:18:57 DEBUG { 23:18:57 DEBUG "metric": { 23:18:57 DEBUG "pod": "ds-idrepo-2" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG [...] 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG "pod": "ds-idrepo-0" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 1713219537, 23:18:57 DEBUG "0" 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG } 23:18:57 INFO [http_cmd]: http status code OK 23:18:57 DEBUG --- status code --- 23:18:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:57 DEBUG --- http response --- 23:18:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:18:57 DEBUG ----- output ----- 23:18:57 DEBUG { 23:18:57 DEBUG "status": "success", 23:18:57 DEBUG "data": { 23:18:57 DEBUG "resultType": "vector", 23:18:57 DEBUG "result": [ 23:18:57 DEBUG { 23:18:57 DEBUG "metric": { 23:18:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG [...] 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 1713219537, 23:18:57 DEBUG "0" 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG } 23:18:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219537" 23:18:57 INFO [http_cmd]: http status code OK 23:18:57 DEBUG --- status code --- 23:18:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:57 DEBUG --- http response --- 23:18:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:18:57 DEBUG ----- output ----- 23:18:57 DEBUG { 23:18:57 DEBUG "status": "success", 23:18:57 DEBUG "data": { 23:18:57 DEBUG "resultType": "vector", 23:18:57 DEBUG "result": [ 23:18:57 DEBUG { 23:18:57 DEBUG "metric": { 23:18:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG [...] 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 1713219537, 23:18:57 DEBUG "6.757627383046117" 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG } 23:18:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219537" 23:18:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219537" 23:18:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219537" 23:18:57 INFO [http_cmd]: http status code OK 23:18:57 DEBUG --- status code --- 23:18:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:57 DEBUG --- http response --- 23:18:57 INFO [http_cmd]: http status code OK 23:18:57 INFO [http_cmd]: http status code OK 23:18:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:57 DEBUG --- status code --- 23:18:57 DEBUG --- status code --- 23:18:57 DEBUG ----- output ----- 23:18:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:18:57 DEBUG { 23:18:57 DEBUG --- http response --- 23:18:57 DEBUG --- http response --- 23:18:57 DEBUG "status": "success", 23:18:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:57 DEBUG "data": { 23:18:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:18:57 DEBUG ----- output ----- 23:18:57 DEBUG "resultType": "vector", 23:18:57 DEBUG ----- output ----- 23:18:57 DEBUG { 23:18:57 DEBUG "result": [ 23:18:57 DEBUG { 23:18:57 DEBUG "status": "success", 23:18:57 DEBUG { 23:18:57 DEBUG "status": "success", 23:18:57 DEBUG "data": { 23:18:57 DEBUG "metric": { 23:18:57 DEBUG "data": { 23:18:57 DEBUG "resultType": "vector", 23:18:57 DEBUG "pod": "ds-idrepo-2" 23:18:57 DEBUG "resultType": "vector", 23:18:57 DEBUG "result": [ 23:18:57 DEBUG }, 23:18:57 DEBUG "result": [ 23:18:57 DEBUG { 23:18:57 DEBUG "value": [ 23:18:57 DEBUG { 23:18:57 DEBUG "metric": { 23:18:57 DEBUG 23:18:57 DEBUG "metric": { 23:18:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:18:57 DEBUG 23:18:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:18:57 DEBUG }, 23:18:57 DEBUG 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG [...] 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG [...] 23:18:57 DEBUG "pod": "ds-idrepo-0" 23:18:57 DEBUG [...] 23:18:57 DEBUG 23:18:57 DEBUG }, 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 23:18:57 DEBUG 23:18:57 DEBUG 1713219537, 23:18:57 DEBUG 23:18:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:18:57 DEBUG "0" 23:18:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:18:57 DEBUG }, 23:18:57 DEBUG ] 23:18:57 DEBUG }, 23:18:57 DEBUG "value": [ 23:18:57 DEBUG } 23:18:57 DEBUG "value": [ 23:18:57 DEBUG 1713219537, 23:18:57 DEBUG ] 23:18:57 DEBUG 1713219537, 23:18:57 DEBUG "2287" 23:18:57 DEBUG } 23:18:57 DEBUG "2290.4" 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG } 23:18:57 DEBUG ] 23:18:57 DEBUG ] 23:18:57 DEBUG } 23:18:57 DEBUG } 23:18:57 DEBUG } 23:18:57 DEBUG } 23:18:57 INFO 23:18:57 INFO [loop_until]: kubectl --namespace=xlou top pods 23:18:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:18:57 INFO [loop_until]: OK (rc = 0) 23:18:57 DEBUG --- stdout --- 23:18:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12878m 8618Mi am-7d597cf7d6-8vr8f 10222m 8895Mi am-7d597cf7d6-9wz9h 15915m 8985Mi ds-cts-0 15223m 8542Mi ds-cts-1 11079m 8603Mi ds-cts-2 10573m 8754Mi ds-idrepo-0 13542m 4977Mi ds-idrepo-1 1796m 2735Mi ds-idrepo-2 1827m 2757Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 1m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6593m 2209Mi 23:18:57 DEBUG --- stderr --- 23:18:57 DEBUG 23:18:57 INFO 23:18:57 INFO [loop_until]: kubectl --namespace=xlou top node 23:18:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:18:57 INFO [loop_until]: OK (rc = 0) 23:18:57 DEBUG --- stdout --- 23:18:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5069Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11037m 36% 10888Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13112m 43% 10669Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17468m 58% 3059Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1965m 6% 5630Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16285m 54% 12880Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2965Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10163m 34% 10398Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1941m 6% 4985Mi 4% gke-xlou-cdm-ds-0f601751-j488 10664m 35% 10401Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2460Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17137m 57% 10633Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13705m 45% 7149Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6840m 43% 6081Mi 10% 23:18:57 DEBUG --- stderr --- 23:18:57 DEBUG 23:19:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219596" 23:19:56 ERROR [http_cmd]: ERROR 23:19:56 ERROR --- status code --- 23:19:56 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:19:56 ERROR --- http response --- 23:19:56 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:19:56 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:19:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219596" 23:19:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219596" 23:19:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219596" 23:19:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219596" 23:19:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219596" 23:19:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219596" 23:19:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219596" 23:19:56 INFO [http_cmd]: http status code OK 23:19:56 INFO [http_cmd]: http status code OK 23:19:56 INFO [http_cmd]: http status code OK 23:19:56 INFO [http_cmd]: http status code OK 23:19:56 DEBUG --- status code --- 23:19:56 INFO [http_cmd]: http status code OK 23:19:56 DEBUG --- status code --- 23:19:56 DEBUG --- status code --- 23:19:56 DEBUG --- status code --- 23:19:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:56 DEBUG --- status code --- 23:19:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:56 DEBUG --- http response --- 23:19:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:56 DEBUG --- http response --- 23:19:56 DEBUG --- http response --- 23:19:56 DEBUG --- http response --- 23:19:56 DEBUG --- http response --- 23:19:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:56 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:19:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:19:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:19:56 DEBUG ----- output ----- 23:19:56 DEBUG ----- output ----- 23:19:56 DEBUG ----- output ----- 23:19:56 DEBUG ----- output ----- 23:19:56 DEBUG { 23:19:56 DEBUG { 23:19:56 DEBUG { 23:19:56 DEBUG { 23:19:56 DEBUG "status": "success", 23:19:56 DEBUG "status": "success", 23:19:56 DEBUG "status": "success", 23:19:56 DEBUG "status": "success", 23:19:56 DEBUG "data": { 23:19:56 DEBUG "data": { 23:19:56 DEBUG "data": { 23:19:56 DEBUG "data": { 23:19:56 DEBUG "resultType": "vector", 23:19:56 DEBUG "resultType": "vector", 23:19:56 DEBUG "resultType": "vector", 23:19:56 INFO [http_cmd]: http status code OK 23:19:56 DEBUG "resultType": "vector", 23:19:56 DEBUG "result": [ 23:19:56 DEBUG "result": [ 23:19:56 DEBUG "result": [ 23:19:56 DEBUG --- status code --- 23:19:56 DEBUG "result": [ 23:19:56 DEBUG { 23:19:56 DEBUG { 23:19:56 DEBUG { 23:19:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:56 DEBUG { 23:19:56 DEBUG "metric": { 23:19:56 DEBUG "metric": { 23:19:56 DEBUG "metric": { 23:19:56 DEBUG --- http response --- 23:19:56 DEBUG "metric": { 23:19:56 DEBUG "pod": "ds-idrepo-2" 23:19:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:19:56 DEBUG "instance": "10.142.0.107:9100" 23:19:56 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:19:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:19:56 DEBUG }, 23:19:56 DEBUG }, 23:19:56 DEBUG }, 23:19:56 DEBUG }, 23:19:56 DEBUG ----- output ----- 23:19:56 DEBUG "value": [ 23:19:56 DEBUG "value": [ 23:19:56 DEBUG "value": [ 23:19:56 DEBUG "value": [ 23:19:56 DEBUG { 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG "status": "success", 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG "data": { 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG "resultType": "vector", 23:19:56 DEBUG [...] 23:19:56 DEBUG [...] 23:19:56 DEBUG [...] 23:19:56 DEBUG [...] 23:19:56 DEBUG "result": [ 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG { 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG "metric": { 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG "pod": "ds-idrepo-2" 23:19:56 DEBUG "pod": "ds-idrepo-0" 23:19:56 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:19:56 DEBUG "instance": "10.142.0.43:9100" 23:19:56 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:19:56 DEBUG }, 23:19:56 DEBUG }, 23:19:56 DEBUG }, 23:19:56 DEBUG }, 23:19:56 DEBUG }, 23:19:56 DEBUG "value": [ 23:19:56 DEBUG "value": [ 23:19:56 DEBUG "value": [ 23:19:56 DEBUG "value": [ 23:19:56 DEBUG "value": [ 23:19:56 DEBUG 23:19:56 DEBUG 1713219596, 23:19:56 DEBUG 1713219596, 23:19:56 DEBUG 1713219596, 23:19:56 DEBUG 1713219596, 23:19:56 DEBUG 23:19:56 DEBUG "0" 23:19:56 DEBUG "0" 23:19:56 DEBUG "12041.93333333333" 23:19:56 DEBUG "10.117353982964103" 23:19:56 DEBUG 23:19:56 DEBUG ] 23:19:56 DEBUG ] 23:19:56 DEBUG ] 23:19:56 DEBUG ] 23:19:56 DEBUG [...] 23:19:56 DEBUG } 23:19:56 DEBUG } 23:19:56 DEBUG } 23:19:56 DEBUG } 23:19:56 DEBUG 23:19:56 DEBUG ] 23:19:56 DEBUG ] 23:19:56 DEBUG ] 23:19:56 DEBUG ] 23:19:56 DEBUG 23:19:56 DEBUG } 23:19:56 DEBUG } 23:19:56 DEBUG } 23:19:56 DEBUG } 23:19:56 DEBUG 23:19:56 DEBUG } 23:19:56 DEBUG } 23:19:56 DEBUG } 23:19:56 DEBUG } 23:19:56 DEBUG "pod": "ds-cts-2" 23:19:56 DEBUG }, 23:19:56 DEBUG "value": [ 23:19:56 DEBUG 1713219596, 23:19:56 DEBUG "0" 23:19:56 DEBUG ] 23:19:56 DEBUG } 23:19:56 DEBUG ] 23:19:56 DEBUG } 23:19:56 DEBUG } 23:19:56 INFO [http_cmd]: http status code OK 23:19:56 DEBUG --- status code --- 23:19:56 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:56 DEBUG --- http response --- 23:19:56 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:56 DEBUG ----- output ----- 23:19:56 DEBUG { 23:19:56 DEBUG "status": "success", 23:19:56 DEBUG "data": { 23:19:56 DEBUG "resultType": "vector", 23:19:56 DEBUG "result": [ 23:19:56 DEBUG { 23:19:56 DEBUG "metric": { 23:19:56 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:19:56 DEBUG }, 23:19:56 DEBUG "value": [ 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG [...] 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG 23:19:56 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:19:56 DEBUG }, 23:19:56 DEBUG "value": [ 23:19:56 DEBUG 1713219596, 23:19:56 DEBUG "2418425856" 23:19:56 DEBUG ] 23:19:56 DEBUG } 23:19:56 DEBUG ] 23:19:56 DEBUG } 23:19:56 DEBUG } 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219597" 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219597" 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG { 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "data": { 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "result": [ 23:19:57 DEBUG { 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "instance": "10.142.0.107:9100" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "instance": "10.142.0.43:9100" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG "0.00040000000000001514" 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG { 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "data": { 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "result": [ 23:19:57 DEBUG { 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "instance": "10.142.0.107:9100" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "instance": "10.142.0.43:9100" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG "3334.0333333333333" 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG { 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "data": { 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "result": [ 23:19:57 DEBUG { 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "pod": "ds-idrepo-2" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "pod": "ds-idrepo-0" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG "0" 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG { 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "data": { 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "result": [ 23:19:57 DEBUG { 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "pod": "ds-idrepo-2" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "pod": "ds-idrepo-0" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG "0" 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219597" 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG { 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "data": { 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "result": [ 23:19:57 DEBUG { 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "pod": "ds-cts-0" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG "40030825.63224948" 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG { 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "data": { 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "result": [ 23:19:57 DEBUG { 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG "2299.8333333333335" 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG { 23:19:57 DEBUG { 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG "data": { 23:19:57 DEBUG "data": { 23:19:57 DEBUG { 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "result": [ 23:19:57 DEBUG "result": [ 23:19:57 DEBUG "data": { 23:19:57 DEBUG { 23:19:57 DEBUG { 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "result": [ 23:19:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:19:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:19:57 DEBUG { 23:19:57 DEBUG }, 23:19:57 DEBUG }, 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "value": [ 23:19:57 DEBUG "value": [ 23:19:57 DEBUG "instance": "10.142.0.107:9100" 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG }, 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:19:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:19:57 DEBUG 23:19:57 DEBUG }, 23:19:57 DEBUG }, 23:19:57 DEBUG 23:19:57 DEBUG "value": [ 23:19:57 DEBUG "value": [ 23:19:57 DEBUG "instance": "10.142.0.43:9100" 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG }, 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG "2298.0666666666666" 23:19:57 DEBUG "0" 23:19:57 DEBUG "value": [ 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG ] 23:19:57 DEBUG ] 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 DEBUG "13903496192" 23:19:57 DEBUG ] 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:57 DEBUG ] 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 DEBUG { 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG } 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG "data": { 23:19:57 DEBUG } 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG "result": [ 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG { 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:57 DEBUG "metric": { 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:19:57 DEBUG { 23:19:57 DEBUG }, 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "value": [ 23:19:57 DEBUG "data": { 23:19:57 DEBUG 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG 23:19:57 DEBUG "result": [ 23:19:57 DEBUG 23:19:57 DEBUG { 23:19:57 DEBUG [...] 23:19:57 DEBUG "metric": { 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "pod": "ds-idrepo-2" 23:19:57 DEBUG 23:19:57 DEBUG }, 23:19:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:19:57 DEBUG "value": [ 23:19:57 DEBUG }, 23:19:57 DEBUG 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG 23:19:57 DEBUG "0.082" 23:19:57 DEBUG [...] 23:19:57 DEBUG ] 23:19:57 DEBUG 23:19:57 DEBUG } 23:19:57 DEBUG 23:19:57 DEBUG ] 23:19:57 DEBUG 23:19:57 DEBUG } 23:19:57 DEBUG "pod": "ds-idrepo-0" 23:19:57 DEBUG } 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG "0" 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG { 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "data": { 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "result": [ 23:19:57 DEBUG { 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG "413629.5" 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219597" 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG { 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "data": { 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "result": [ 23:19:57 DEBUG { 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "instance": "10.142.0.107:9100" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "instance": "10.142.0.43:9100" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG "0.0003333333333333262" 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219597" 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG { 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG { 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG "data": { 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:19:57 DEBUG "data": { 23:19:57 DEBUG "result": [ 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG { 23:19:57 DEBUG { 23:19:57 DEBUG "result": [ 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG { 23:19:57 DEBUG "pod": "ds-idrepo-2" 23:19:57 DEBUG "data": { 23:19:57 DEBUG "metric": { 23:19:57 DEBUG }, 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:19:57 DEBUG "value": [ 23:19:57 DEBUG "result": [ 23:19:57 DEBUG }, 23:19:57 DEBUG 23:19:57 DEBUG { 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG "metric": { 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG }, 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "value": [ 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "pod": "ds-idrepo-0" 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG }, 23:19:57 DEBUG [...] 23:19:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG }, 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG 23:19:57 DEBUG "value": [ 23:19:57 DEBUG "0" 23:19:57 DEBUG 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG ] 23:19:57 DEBUG "pod": "ds-cts-0" 23:19:57 DEBUG "6.70364473480405" 23:19:57 DEBUG } 23:19:57 DEBUG }, 23:19:57 DEBUG ] 23:19:57 DEBUG ] 23:19:57 DEBUG "value": [ 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG "283836807.50459534" 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219597" 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 INFO [http_cmd]: http status code OK 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG --- status code --- 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG --- http response --- 23:19:57 DEBUG { 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG "data": { 23:19:57 DEBUG ----- output ----- 23:19:57 DEBUG { 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG { 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "result": [ 23:19:57 DEBUG "status": "success", 23:19:57 DEBUG "data": { 23:19:57 DEBUG { 23:19:57 DEBUG "data": { 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "resultType": "vector", 23:19:57 DEBUG "result": [ 23:19:57 DEBUG "pod": "ds-idrepo-2" 23:19:57 DEBUG "result": [ 23:19:57 DEBUG { 23:19:57 DEBUG }, 23:19:57 DEBUG { 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "value": [ 23:19:57 DEBUG "metric": { 23:19:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:19:57 DEBUG 23:19:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:19:57 DEBUG }, 23:19:57 DEBUG 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG [...] 23:19:57 DEBUG 23:19:57 DEBUG "pod": "ds-idrepo-0" 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG }, 23:19:57 DEBUG 23:19:57 DEBUG 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 23:19:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:19:57 DEBUG }, 23:19:57 DEBUG "0" 23:19:57 DEBUG }, 23:19:57 DEBUG "value": [ 23:19:57 DEBUG ] 23:19:57 DEBUG "value": [ 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG } 23:19:57 DEBUG 1713219597, 23:19:57 DEBUG "2296.5" 23:19:57 DEBUG ] 23:19:57 DEBUG "2288.1" 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 DEBUG ] 23:19:57 DEBUG ] 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 DEBUG } 23:19:57 INFO 23:19:57 INFO [loop_until]: kubectl --namespace=xlou top pods 23:19:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:19:57 INFO [loop_until]: OK (rc = 0) 23:19:57 DEBUG --- stdout --- 23:19:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13072m 8613Mi am-7d597cf7d6-8vr8f 10044m 8894Mi am-7d597cf7d6-9wz9h 15774m 9035Mi ds-cts-0 15148m 8551Mi ds-cts-1 10801m 8631Mi ds-cts-2 10470m 8776Mi ds-idrepo-0 13265m 4977Mi ds-idrepo-1 1802m 2734Mi ds-idrepo-2 1830m 2757Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6762m 2216Mi 23:19:57 DEBUG --- stderr --- 23:19:57 DEBUG 23:19:57 INFO 23:19:57 INFO [loop_until]: kubectl --namespace=xlou top node 23:19:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:19:57 INFO [loop_until]: OK (rc = 0) 23:19:57 DEBUG --- stdout --- 23:19:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5076Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11022m 36% 10895Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13168m 44% 10677Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17551m 58% 3072Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1970m 6% 5628Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16081m 53% 12865Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2974Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10271m 34% 10405Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1928m 6% 4991Mi 4% gke-xlou-cdm-ds-0f601751-j488 10198m 34% 10415Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2457Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17246m 57% 10689Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13851m 46% 7156Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6872m 43% 6084Mi 10% 23:19:57 DEBUG --- stderr --- 23:19:57 DEBUG 23:20:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219656" 23:20:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219656" 23:20:56 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219656" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 ERROR [http_cmd]: ERROR 23:20:57 ERROR --- status code --- 23:20:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:20:57 ERROR --- http response --- 23:20:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:20:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "instance": "10.142.0.107:9100" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "instance": "10.142.0.43:9100" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219656, 23:20:57 DEBUG "12335.166666666664" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "ds-idrepo-2" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219656, 23:20:57 DEBUG "0.9885424298836277" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "ds-idrepo-2" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "ds-idrepo-0" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "0" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "ds-idrepo-2" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "4009.322356608056" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219657" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "ds-idrepo-2" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "ds-cts-2" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "0" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219657" 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG { 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG "metric": { 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:20:57 DEBUG [...] 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG 23:20:57 DEBUG { 23:20:57 DEBUG 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG 23:20:57 DEBUG "data": { 23:20:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG }, 23:20:57 DEBUG "result": [ 23:20:57 DEBUG "value": [ 23:20:57 DEBUG { 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "2424766464" 23:20:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:20:57 DEBUG ] 23:20:57 DEBUG }, 23:20:57 DEBUG } 23:20:57 DEBUG "value": [ 23:20:57 DEBUG ] 23:20:57 DEBUG 23:20:57 DEBUG } 23:20:57 DEBUG 23:20:57 DEBUG } 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "ds-idrepo-2" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "3142.441847071179" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "instance": "10.142.0.107:9100" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "instance": "10.142.0.43:9100" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "0.0008333333333330491" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219657" 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "instance": "10.142.0.107:9100" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "instance": "10.142.0.43:9100" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "3091.366666666667" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "ds-idrepo-2" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "ds-idrepo-0" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "0" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219657" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG { 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG "instance": "10.142.0.107:9100" 23:20:57 DEBUG "pod": "ds-idrepo-2" 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG }, 23:20:57 DEBUG }, 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG "value": [ 23:20:57 DEBUG "value": [ 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:57 DEBUG 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:20:57 DEBUG 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG 23:20:57 DEBUG { 23:20:57 DEBUG 23:20:57 DEBUG { 23:20:57 DEBUG 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG [...] 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG [...] 23:20:57 DEBUG "data": { 23:20:57 DEBUG 23:20:57 DEBUG "data": { 23:20:57 DEBUG 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG 23:20:57 DEBUG "resultType": "vector", 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 DEBUG 23:20:57 DEBUG "result": [ 23:20:57 DEBUG 23:20:57 DEBUG "result": [ 23:20:57 DEBUG 23:20:57 DEBUG { 23:20:57 DEBUG "instance": "10.142.0.43:9100" 23:20:57 DEBUG { 23:20:57 DEBUG "pod": "ds-idrepo-0" 23:20:57 DEBUG "metric": { 23:20:57 DEBUG }, 23:20:57 DEBUG "metric": { 23:20:57 DEBUG }, 23:20:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:20:57 DEBUG "value": [ 23:20:57 DEBUG "pod": "ds-idrepo-2" 23:20:57 DEBUG "value": [ 23:20:57 DEBUG }, 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG }, 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG "13904178176" 23:20:57 DEBUG "value": [ 23:20:57 DEBUG "0" 23:20:57 DEBUG 23:20:57 DEBUG ] 23:20:57 DEBUG 23:20:57 DEBUG ] 23:20:57 DEBUG 23:20:57 DEBUG } 23:20:57 DEBUG 23:20:57 DEBUG } 23:20:57 DEBUG 23:20:57 DEBUG ] 23:20:57 DEBUG 23:20:57 DEBUG ] 23:20:57 DEBUG [...] 23:20:57 DEBUG } 23:20:57 DEBUG [...] 23:20:57 DEBUG } 23:20:57 DEBUG 23:20:57 DEBUG } 23:20:57 DEBUG 23:20:57 DEBUG } 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:20:57 DEBUG "pod": "ds-idrepo-0" 23:20:57 DEBUG }, 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "2289.2" 23:20:57 DEBUG "0" 23:20:57 DEBUG ] 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "2287.133333333333" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "0" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "0.082" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "413105.5" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219657" 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "instance": "10.142.0.107:9100" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "instance": "10.142.0.43:9100" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "0" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219657" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "ds-idrepo-2" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG [...] 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:20:57 DEBUG "pod": "ds-idrepo-0" 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG }, 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "value": [ 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "0" 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG ] 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG } 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG ] 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG } 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:20:57 DEBUG } 23:20:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:20:57 DEBUG }, 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "ds-idrepo-2" 23:20:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:20:57 DEBUG }, 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "7260.124267371344" 23:20:57 DEBUG "6.71475235384471" 23:20:57 DEBUG ] 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219657" 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG { 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "ds-idrepo-2" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "ds-idrepo-0" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "0" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "2298.0333333333333" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO [http_cmd]: http status code OK 23:20:57 DEBUG --- status code --- 23:20:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:20:57 DEBUG --- http response --- 23:20:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:20:57 DEBUG ----- output ----- 23:20:57 DEBUG { 23:20:57 DEBUG "status": "success", 23:20:57 DEBUG "data": { 23:20:57 DEBUG "resultType": "vector", 23:20:57 DEBUG "result": [ 23:20:57 DEBUG { 23:20:57 DEBUG "metric": { 23:20:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG [...] 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG 23:20:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:20:57 DEBUG }, 23:20:57 DEBUG "value": [ 23:20:57 DEBUG 1713219657, 23:20:57 DEBUG "2284.733333333333" 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG ] 23:20:57 DEBUG } 23:20:57 DEBUG } 23:20:57 INFO 23:20:57 INFO [loop_until]: kubectl --namespace=xlou top pods 23:20:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:20:57 INFO [loop_until]: OK (rc = 0) 23:20:57 DEBUG --- stdout --- 23:20:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12791m 8654Mi am-7d597cf7d6-8vr8f 10085m 8893Mi am-7d597cf7d6-9wz9h 16070m 9030Mi ds-cts-0 15622m 8511Mi ds-cts-1 10671m 8618Mi ds-cts-2 10730m 8758Mi ds-idrepo-0 13314m 4977Mi ds-idrepo-1 1829m 2735Mi ds-idrepo-2 1818m 2756Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6506m 2219Mi 23:20:57 DEBUG --- stderr --- 23:20:57 DEBUG 23:20:57 INFO 23:20:57 INFO [loop_until]: kubectl --namespace=xlou top node 23:20:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:20:57 INFO [loop_until]: OK (rc = 0) 23:20:57 DEBUG --- stdout --- 23:20:57 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5079Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10864m 36% 10897Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13233m 44% 10728Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17027m 57% 3076Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1973m 6% 5623Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16628m 55% 12880Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2978Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10221m 34% 10401Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1923m 6% 4986Mi 4% gke-xlou-cdm-ds-0f601751-j488 10770m 36% 10414Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2456Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17382m 58% 10677Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13734m 46% 7154Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6858m 43% 6087Mi 10% 23:20:57 DEBUG --- stderr --- 23:20:57 DEBUG 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219717" 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "pod": "ds-idrepo-2" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "pod": "ds-idrepo-0" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "0" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 ERROR [http_cmd]: ERROR 23:21:57 ERROR --- status code --- 23:21:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:21:57 ERROR --- http response --- 23:21:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:21:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "pod": "ds-idrepo-2" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "pod": "ds-cts-2" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "0" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG { 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG "data": { 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG "result": [ 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG { 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG }, 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG "value": [ 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:21:57 DEBUG 23:21:57 DEBUG "instance": "10.142.0.107:9100" 23:21:57 DEBUG }, 23:21:57 DEBUG [...] 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "pod": "ds-idrepo-2" 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG }, 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "5512.826565991217" 23:21:57 DEBUG 23:21:57 DEBUG "pod": "ds-idrepo-2" 23:21:57 DEBUG ] 23:21:57 DEBUG "instance": "10.142.0.43:9100" 23:21:57 DEBUG }, 23:21:57 DEBUG } 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG ] 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG } 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "1.3046012162339036" 23:21:57 DEBUG } 23:21:57 DEBUG "12223.999999999998" 23:21:57 DEBUG ] 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219717" 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "2428702720" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219717" 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "pod": "ds-idrepo-2" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "4050.239926034363" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "instance": "10.142.0.107:9100" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "instance": "10.142.0.43:9100" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "0.00043333333333350765" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "instance": "10.142.0.107:9100" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "instance": "10.142.0.43:9100" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "4364" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "pod": "ds-idrepo-2" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "pod": "ds-idrepo-0" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "0" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219717" 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "instance": "10.142.0.107:9100" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "instance": "10.142.0.43:9100" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "13904839680" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "pod": "ds-idrepo-2" 23:21:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:21:57 DEBUG }, 23:21:57 DEBUG "pod": "ds-idrepo-2" 23:21:57 DEBUG }, 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG [...] 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG [...] 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:57 DEBUG 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG "pod": "ds-idrepo-0" 23:21:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:21:57 DEBUG }, 23:21:57 DEBUG { 23:21:57 DEBUG "pod": "ds-idrepo-0" 23:21:57 DEBUG { 23:21:57 DEBUG }, 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG }, 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "value": [ 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "data": { 23:21:57 DEBUG "value": [ 23:21:57 DEBUG "data": { 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "412502" 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "0" 23:21:57 DEBUG "2306.7666666666664" 23:21:57 DEBUG ] 23:21:57 DEBUG "result": [ 23:21:57 DEBUG "0" 23:21:57 DEBUG "result": [ 23:21:57 DEBUG ] 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG { 23:21:57 DEBUG ] 23:21:57 DEBUG { 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG "metric": { 23:21:57 DEBUG } 23:21:57 DEBUG "metric": { 23:21:57 DEBUG ] 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:21:57 DEBUG ] 23:21:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 DEBUG }, 23:21:57 DEBUG } 23:21:57 DEBUG }, 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 DEBUG "value": [ 23:21:57 DEBUG } 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:21:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:21:57 DEBUG }, 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG "0" 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG ] 23:21:57 DEBUG "2308.0666666666666" 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "0.082" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219717" 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "instance": "10.142.0.107:9100" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "instance": "10.142.0.43:9100" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "0.000333333333333341" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "pod": "ds-idrepo-2" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "pod": "ds-idrepo-0" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "0" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219717" 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "pod": "ds-idrepo-2" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "7762.95985823253" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "6.816874167074362" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219717" 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "pod": "ds-idrepo-2" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "pod": "ds-idrepo-0" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "0" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "2285.3333333333335" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO [http_cmd]: http status code OK 23:21:57 DEBUG --- status code --- 23:21:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:21:57 DEBUG --- http response --- 23:21:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:21:57 DEBUG ----- output ----- 23:21:57 DEBUG { 23:21:57 DEBUG "status": "success", 23:21:57 DEBUG "data": { 23:21:57 DEBUG "resultType": "vector", 23:21:57 DEBUG "result": [ 23:21:57 DEBUG { 23:21:57 DEBUG "metric": { 23:21:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG [...] 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG 23:21:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:21:57 DEBUG }, 23:21:57 DEBUG "value": [ 23:21:57 DEBUG 1713219717, 23:21:57 DEBUG "2302.266666666667" 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG ] 23:21:57 DEBUG } 23:21:57 DEBUG } 23:21:57 INFO 23:21:57 INFO [loop_until]: kubectl --namespace=xlou top pods 23:21:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:21:57 INFO [loop_until]: OK (rc = 0) 23:21:57 DEBUG --- stdout --- 23:21:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12806m 8628Mi am-7d597cf7d6-8vr8f 10039m 8895Mi am-7d597cf7d6-9wz9h 16039m 8990Mi ds-cts-0 15602m 8539Mi ds-cts-1 10898m 8623Mi ds-cts-2 11015m 8769Mi ds-idrepo-0 13411m 4977Mi ds-idrepo-1 1820m 2735Mi ds-idrepo-2 1813m 2757Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 1m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6592m 2221Mi 23:21:57 DEBUG --- stderr --- 23:21:57 DEBUG 23:21:57 INFO 23:21:57 INFO [loop_until]: kubectl --namespace=xlou top node 23:21:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:21:58 INFO [loop_until]: OK (rc = 0) 23:21:58 DEBUG --- stdout --- 23:21:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5075Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10887m 36% 10894Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13177m 44% 10710Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17606m 58% 3062Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1978m 6% 5634Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16443m 55% 12883Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2964Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 9909m 33% 10390Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1908m 6% 4984Mi 4% gke-xlou-cdm-ds-0f601751-j488 10970m 36% 10420Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2447Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17401m 58% 10646Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13826m 46% 7167Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6873m 43% 6091Mi 10% 23:21:58 DEBUG --- stderr --- 23:21:58 DEBUG 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "ds-idrepo-2" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "ds-idrepo-0" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "0" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "0" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "instance": "10.142.0.107:9100" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "instance": "10.142.0.43:9100" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "12306.499999999998" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "6.65736826610731" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 ERROR [http_cmd]: ERROR 23:22:57 ERROR --- status code --- 23:22:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:22:57 ERROR --- http response --- 23:22:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:22:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "ds-idrepo-2" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "ds-cts-2" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "0" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219777" 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "instance": "10.142.0.107:9100" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "instance": "10.142.0.43:9100" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "3257.733333333333" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "ds-idrepo-2" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "ds-idrepo-0" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "0" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "2428612608" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219777" 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "0" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "instance": "10.142.0.107:9100" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "instance": "10.142.0.43:9100" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "0.00043333333333303397" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "instance": "10.142.0.107:9100" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "instance": "10.142.0.43:9100" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "13905470464" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "ds-idrepo-2" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "ds-idrepo-0" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "0" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219777" 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:22:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:22:57 DEBUG }, 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:22:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:22:57 DEBUG }, 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "0" 23:22:57 DEBUG "2294.7666666666664" 23:22:57 DEBUG ] 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "ds-idrepo-2" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "ds-idrepo-0" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "0" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "412679.5" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "2294.9333333333334" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "instance": "10.142.0.107:9100" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "instance": "10.142.0.43:9100" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "0.00033333333333333365" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "0.082" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "ds-idrepo-2" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "ds-idrepo-0" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "0" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219777" 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "80586237.52246648" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "6.676174652826671" 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219777" 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 INFO [http_cmd]: http status code OK 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG --- status code --- 23:22:57 DEBUG { 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG --- http response --- 23:22:57 DEBUG "data": { 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG "result": [ 23:22:57 DEBUG ----- output ----- 23:22:57 DEBUG { 23:22:57 DEBUG { 23:22:57 DEBUG { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "status": "success", 23:22:57 DEBUG "data": { 23:22:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:22:57 DEBUG "data": { 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG }, 23:22:57 DEBUG "resultType": "vector", 23:22:57 DEBUG "result": [ 23:22:57 DEBUG "value": [ 23:22:57 DEBUG "result": [ 23:22:57 DEBUG { 23:22:57 DEBUG 23:22:57 DEBUG { 23:22:57 DEBUG "metric": { 23:22:57 DEBUG 23:22:57 DEBUG "metric": { 23:22:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:22:57 DEBUG 23:22:57 DEBUG "pod": "ds-idrepo-2" 23:22:57 DEBUG }, 23:22:57 DEBUG [...] 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:22:57 DEBUG 23:22:57 DEBUG [...] 23:22:57 DEBUG }, 23:22:57 DEBUG [...] 23:22:57 DEBUG 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG 23:22:57 DEBUG 23:22:57 DEBUG "2300.133333333333" 23:22:57 DEBUG 23:22:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:22:57 DEBUG ] 23:22:57 DEBUG "pod": "ds-idrepo-0" 23:22:57 DEBUG }, 23:22:57 DEBUG } 23:22:57 DEBUG }, 23:22:57 DEBUG "value": [ 23:22:57 DEBUG ] 23:22:57 DEBUG "value": [ 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG } 23:22:57 DEBUG 1713219777, 23:22:57 DEBUG "2294.8" 23:22:57 DEBUG } 23:22:57 DEBUG "0" 23:22:57 DEBUG ] 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 DEBUG ] 23:22:57 DEBUG ] 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 DEBUG } 23:22:57 INFO 23:22:57 INFO [loop_until]: kubectl --namespace=xlou top pods 23:22:57 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:22:57 INFO [loop_until]: OK (rc = 0) 23:22:57 DEBUG --- stdout --- 23:22:57 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12820m 8633Mi am-7d597cf7d6-8vr8f 10054m 8895Mi am-7d597cf7d6-9wz9h 16158m 9019Mi ds-cts-0 15102m 8544Mi ds-cts-1 10726m 8620Mi ds-cts-2 10401m 8763Mi ds-idrepo-0 13345m 4983Mi ds-idrepo-1 1824m 2735Mi ds-idrepo-2 1756m 2757Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6645m 2222Mi 23:22:57 DEBUG --- stderr --- 23:22:57 DEBUG 23:22:58 INFO 23:22:58 INFO [loop_until]: kubectl --namespace=xlou top node 23:22:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:22:58 INFO [loop_until]: OK (rc = 0) 23:22:58 DEBUG --- stdout --- 23:22:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5078Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11037m 36% 10895Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13136m 43% 10707Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17569m 58% 3082Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1973m 6% 5625Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16422m 55% 12893Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2960Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10267m 34% 10400Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1893m 6% 4985Mi 4% gke-xlou-cdm-ds-0f601751-j488 10747m 35% 10428Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2458Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17251m 57% 10670Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13292m 44% 7158Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6834m 43% 6090Mi 10% 23:22:58 DEBUG --- stderr --- 23:22:58 DEBUG 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "ds-idrepo-2" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "ds-idrepo-0" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "0" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219837" 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 ERROR [http_cmd]: ERROR 23:23:57 DEBUG "resultType": "vector", 23:23:57 ERROR --- status code --- 23:23:57 DEBUG "result": [ 23:23:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:23:57 DEBUG { 23:23:57 ERROR --- http response --- 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219837" 23:23:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:23:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "0" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "ds-idrepo-2" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "ds-cts-2" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "0" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:23:57 DEBUG "instance": "10.142.0.107:9100" 23:23:57 DEBUG }, 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:23:57 DEBUG "instance": "10.142.0.43:9100" 23:23:57 DEBUG }, 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "3.2457425546431504" 23:23:57 DEBUG "12428.8" 23:23:57 DEBUG ] 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219837" 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219837" 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "instance": "10.142.0.107:9100" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "instance": "10.142.0.43:9100" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "13906369536" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "instance": "10.142.0.107:9100" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "instance": "10.142.0.43:9100" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "4548.966666666666" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "0" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "2429726720" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "ds-idrepo-2" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "ds-idrepo-0" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "0" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "ds-idrepo-2" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "ds-idrepo-0" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "0" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219837" 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "instance": "10.142.0.107:9100" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "instance": "10.142.0.43:9100" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "0.00036666666666652263" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "2298.8" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "0" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "2298.9" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "instance": "10.142.0.107:9100" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "instance": "10.142.0.43:9100" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "0.0003333333333333558" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "ds-idrepo-2" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "ds-idrepo-0" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "0" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "413367" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "0.082" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "ds-idrepo-2" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "ds-idrepo-0" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "0" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "66472479.95511922" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "6.720739495650162" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219837" 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG { 23:23:57 DEBUG { 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG "data": { 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "ds-idrepo-2" 23:23:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:23:57 DEBUG }, 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "ds-idrepo-0" 23:23:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:23:57 DEBUG }, 23:23:57 INFO [http_cmd]: http status code OK 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG --- status code --- 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "0" 23:23:57 DEBUG --- http response --- 23:23:57 DEBUG "2345.866666666667" 23:23:57 DEBUG ] 23:23:57 DEBUG ] 23:23:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 DEBUG ----- output ----- 23:23:57 DEBUG ] 23:23:57 DEBUG ] 23:23:57 DEBUG { 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 DEBUG "status": "success", 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:57 DEBUG "data": { 23:23:57 DEBUG "resultType": "vector", 23:23:57 DEBUG "result": [ 23:23:57 DEBUG { 23:23:57 DEBUG "metric": { 23:23:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG [...] 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG 23:23:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:23:57 DEBUG }, 23:23:57 DEBUG "value": [ 23:23:57 DEBUG 1713219837, 23:23:57 DEBUG "2303.633333333333" 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG ] 23:23:57 DEBUG } 23:23:57 DEBUG } 23:23:58 INFO 23:23:58 INFO [loop_until]: kubectl --namespace=xlou top pods 23:23:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:23:58 INFO [loop_until]: OK (rc = 0) 23:23:58 DEBUG --- stdout --- 23:23:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12756m 8637Mi am-7d597cf7d6-8vr8f 10111m 8894Mi am-7d597cf7d6-9wz9h 16081m 9000Mi ds-cts-0 15558m 8526Mi ds-cts-1 10966m 8616Mi ds-cts-2 10575m 8774Mi ds-idrepo-0 13432m 4984Mi ds-idrepo-1 1821m 2746Mi ds-idrepo-2 1815m 2758Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1026Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6633m 2226Mi 23:23:58 DEBUG --- stderr --- 23:23:58 DEBUG 23:23:58 INFO 23:23:58 INFO [loop_until]: kubectl --namespace=xlou top node 23:23:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:23:58 INFO [loop_until]: OK (rc = 0) 23:23:58 DEBUG --- stdout --- 23:23:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5078Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11110m 37% 10912Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13230m 44% 10711Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17715m 59% 3080Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1908m 6% 5632Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16991m 56% 12881Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2964Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10309m 34% 10401Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1958m 6% 5001Mi 4% gke-xlou-cdm-ds-0f601751-j488 10800m 36% 10426Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2451Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17277m 57% 10651Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13901m 46% 7167Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6899m 43% 6085Mi 10% 23:23:58 DEBUG --- stderr --- 23:23:58 DEBUG 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "ds-idrepo-2" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "ds-idrepo-0" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "0" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "0" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219897" 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "ds-idrepo-2" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "ds-cts-2" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "0" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 ERROR [http_cmd]: ERROR 23:24:57 ERROR --- status code --- 23:24:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:24:57 ERROR --- http response --- 23:24:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:24:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219897" 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:24:57 DEBUG "instance": "10.142.0.107:9100" 23:24:57 DEBUG }, 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:24:57 DEBUG "instance": "10.142.0.43:9100" 23:24:57 DEBUG }, 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "3.7181353703220674" 23:24:57 DEBUG "12050.499999999998" 23:24:57 DEBUG ] 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219897" 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219897" 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219897" 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "ds-idrepo-2" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "ds-idrepo-0" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "0" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "instance": "10.142.0.107:9100" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "instance": "10.142.0.43:9100" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "13906887680" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219897" 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:24:57 DEBUG "instance": "10.142.0.107:9100" 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG }, 23:24:57 DEBUG { 23:24:57 DEBUG "value": [ 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG 23:24:57 DEBUG "data": { 23:24:57 DEBUG 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG 23:24:57 DEBUG "result": [ 23:24:57 DEBUG [...] 23:24:57 DEBUG { 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG 23:24:57 DEBUG "metric": { 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG 23:24:57 DEBUG "instance": "10.142.0.107:9100" 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG 23:24:57 DEBUG }, 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG "instance": "10.142.0.43:9100" 23:24:57 DEBUG "value": [ 23:24:57 DEBUG }, 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:24:57 DEBUG 23:24:57 DEBUG "value": [ 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG { 23:24:57 DEBUG 23:24:57 DEBUG "3244.2999999999997" 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG [...] 23:24:57 DEBUG ] 23:24:57 DEBUG "data": { 23:24:57 DEBUG 23:24:57 DEBUG } 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG 23:24:57 DEBUG ] 23:24:57 DEBUG "result": [ 23:24:57 DEBUG 23:24:57 DEBUG } 23:24:57 DEBUG { 23:24:57 DEBUG "instance": "10.142.0.43:9100" 23:24:57 DEBUG } 23:24:57 DEBUG "metric": { 23:24:57 DEBUG }, 23:24:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:24:57 DEBUG "value": [ 23:24:57 DEBUG }, 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG "0.00033333333333350387" 23:24:57 DEBUG 23:24:57 DEBUG ] 23:24:57 DEBUG 23:24:57 DEBUG } 23:24:57 DEBUG 23:24:57 DEBUG ] 23:24:57 DEBUG [...] 23:24:57 DEBUG } 23:24:57 DEBUG 23:24:57 DEBUG } 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "0" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "2432860160" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "ds-idrepo-2" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "ds-idrepo-0" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "0" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219897" 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "2281.4" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "instance": "10.142.0.107:9100" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "instance": "10.142.0.43:9100" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "0" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "2281.233333333333" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "0" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "413145" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "ds-idrepo-2" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "ds-idrepo-0" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "0" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "0.082" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "ds-idrepo-2" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "ds-idrepo-0" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "0" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219897" 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "26197.84293467692" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "6.7615099065807875" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "2285.4666666666667" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "ds-idrepo-2" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "ds-idrepo-0" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "0" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219897" 23:24:57 INFO [http_cmd]: http status code OK 23:24:57 DEBUG --- status code --- 23:24:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:24:57 DEBUG --- http response --- 23:24:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:24:57 DEBUG ----- output ----- 23:24:57 DEBUG { 23:24:57 DEBUG "status": "success", 23:24:57 DEBUG "data": { 23:24:57 DEBUG "resultType": "vector", 23:24:57 DEBUG "result": [ 23:24:57 DEBUG { 23:24:57 DEBUG "metric": { 23:24:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG [...] 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG 23:24:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:24:57 DEBUG }, 23:24:57 DEBUG "value": [ 23:24:57 DEBUG 1713219897, 23:24:57 DEBUG "2307.7" 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG ] 23:24:57 DEBUG } 23:24:57 DEBUG } 23:24:58 INFO 23:24:58 INFO [loop_until]: kubectl --namespace=xlou top pods 23:24:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:24:58 INFO [loop_until]: OK (rc = 0) 23:24:58 DEBUG --- stdout --- 23:24:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12636m 8632Mi am-7d597cf7d6-8vr8f 10151m 8897Mi am-7d597cf7d6-9wz9h 16140m 9016Mi ds-cts-0 15597m 8571Mi ds-cts-1 11128m 8633Mi ds-cts-2 11426m 8767Mi ds-idrepo-0 13234m 4984Mi ds-idrepo-1 1816m 2746Mi ds-idrepo-2 1877m 2759Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6641m 2226Mi 23:24:58 DEBUG --- stderr --- 23:24:58 DEBUG 23:24:58 INFO 23:24:58 INFO [loop_until]: kubectl --namespace=xlou top node 23:24:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:24:58 INFO [loop_until]: OK (rc = 0) 23:24:58 DEBUG --- stdout --- 23:24:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5078Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10946m 36% 10913Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13131m 43% 10702Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 16961m 56% 3065Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1968m 6% 5644Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16032m 53% 12884Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2972Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10293m 34% 10403Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1919m 6% 5002Mi 4% gke-xlou-cdm-ds-0f601751-j488 10832m 36% 10431Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2439Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17201m 57% 10671Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13791m 46% 7162Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6847m 43% 6087Mi 10% 23:24:58 DEBUG --- stderr --- 23:24:58 DEBUG 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713219957" 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219957" 23:25:57 ERROR [http_cmd]: ERROR 23:25:57 ERROR --- status code --- 23:25:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 ERROR --- http response --- 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "ds-idrepo-2" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "ds-idrepo-0" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "instance": "10.142.0.107:9100" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "instance": "10.142.0.43:9100" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "11971.766666666665" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "ds-idrepo-2" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "ds-cts-2" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219957" 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "instance": "10.142.0.107:9100" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "instance": "10.142.0.43:9100" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "13907575808" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713219957" 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219957" 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "2433679360" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "instance": "10.142.0.107:9100" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "instance": "10.142.0.43:9100" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0.00040000000000001514" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219957" 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "ds-idrepo-2" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "ds-idrepo-0" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "instance": "10.142.0.107:9100" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:25:57 DEBUG 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG 23:25:57 DEBUG { 23:25:57 DEBUG "instance": "10.142.0.43:9100" 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG }, 23:25:57 DEBUG "data": { 23:25:57 DEBUG "value": [ 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "result": [ 23:25:57 DEBUG "3038.0666666666666" 23:25:57 DEBUG { 23:25:57 DEBUG ] 23:25:57 DEBUG "metric": { 23:25:57 DEBUG } 23:25:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:25:57 DEBUG ] 23:25:57 DEBUG }, 23:25:57 DEBUG } 23:25:57 DEBUG "value": [ 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 DEBUG } 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "830.3710656381361" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "2285.3" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "ds-idrepo-2" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "ds-idrepo-0" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713219957" 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "2285.4666666666667" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "instance": "10.142.0.107:9100" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "instance": "10.142.0.43:9100" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "ds-idrepo-2" 23:25:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:25:57 DEBUG }, 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "ds-idrepo-0" 23:25:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:25:57 DEBUG }, 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0" 23:25:57 DEBUG "412275" 23:25:57 DEBUG ] 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "ds-idrepo-2" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "ds-idrepo-0" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0.082" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713219957" 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "6.67701078967342" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "2290.9333333333334" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "ds-idrepo-2" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "ds-idrepo-0" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "0" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713219957" 23:25:57 INFO [http_cmd]: http status code OK 23:25:57 DEBUG --- status code --- 23:25:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:25:57 DEBUG --- http response --- 23:25:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:25:57 DEBUG ----- output ----- 23:25:57 DEBUG { 23:25:57 DEBUG "status": "success", 23:25:57 DEBUG "data": { 23:25:57 DEBUG "resultType": "vector", 23:25:57 DEBUG "result": [ 23:25:57 DEBUG { 23:25:57 DEBUG "metric": { 23:25:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG [...] 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG 23:25:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:25:57 DEBUG }, 23:25:57 DEBUG "value": [ 23:25:57 DEBUG 1713219957, 23:25:57 DEBUG "2294.366666666667" 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG ] 23:25:57 DEBUG } 23:25:57 DEBUG } 23:25:58 INFO 23:25:58 INFO [loop_until]: kubectl --namespace=xlou top pods 23:25:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:25:58 INFO [loop_until]: OK (rc = 0) 23:25:58 DEBUG --- stdout --- 23:25:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12841m 8683Mi am-7d597cf7d6-8vr8f 10191m 8896Mi am-7d597cf7d6-9wz9h 15913m 8994Mi ds-cts-0 15332m 8547Mi ds-cts-1 11093m 8646Mi ds-cts-2 10593m 8780Mi ds-idrepo-0 13480m 4985Mi ds-idrepo-1 1814m 2746Mi ds-idrepo-2 1847m 2758Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6698m 2229Mi 23:25:58 DEBUG --- stderr --- 23:25:58 DEBUG 23:25:58 INFO 23:25:58 INFO [loop_until]: kubectl --namespace=xlou top node 23:25:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:25:58 INFO [loop_until]: OK (rc = 0) 23:25:58 DEBUG --- stdout --- 23:25:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5076Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11058m 37% 10917Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13057m 43% 10751Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17529m 58% 3065Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1988m 6% 5646Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16315m 54% 12882Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2989Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10343m 34% 10397Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1937m 6% 5005Mi 4% gke-xlou-cdm-ds-0f601751-j488 10761m 36% 10441Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2440Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16619m 55% 10654Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13791m 46% 7168Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6856m 43% 6100Mi 10% 23:25:58 DEBUG --- stderr --- 23:25:58 DEBUG 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 ERROR [http_cmd]: ERROR 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 ERROR --- status code --- 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG --- status code --- 23:26:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 ERROR --- http response --- 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:26:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG ----- output ----- 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG { 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG { 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG "data": { 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG "data": { 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG "result": [ 23:26:57 DEBUG "result": [ 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG { 23:26:57 DEBUG { 23:26:57 DEBUG { 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:26:57 DEBUG "pod": "ds-idrepo-2" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220017" 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:26:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:26:57 DEBUG "data": { 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG }, 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG }, 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG { 23:26:57 DEBUG "value": [ 23:26:57 DEBUG { 23:26:57 DEBUG "value": [ 23:26:57 DEBUG "result": [ 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG 23:26:57 DEBUG { 23:26:57 DEBUG "data": { 23:26:57 DEBUG 23:26:57 DEBUG "data": { 23:26:57 DEBUG 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG 23:26:57 DEBUG "instance": "10.142.0.107:9100" 23:26:57 DEBUG "result": [ 23:26:57 DEBUG [...] 23:26:57 DEBUG "result": [ 23:26:57 DEBUG [...] 23:26:57 DEBUG }, 23:26:57 DEBUG { 23:26:57 DEBUG 23:26:57 DEBUG { 23:26:57 DEBUG 23:26:57 DEBUG "value": [ 23:26:57 DEBUG "metric": { 23:26:57 DEBUG 23:26:57 DEBUG "metric": { 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:26:57 DEBUG 23:26:57 DEBUG "pod": "ds-idrepo-2" 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG }, 23:26:57 DEBUG "pod": "ds-idrepo-0" 23:26:57 DEBUG }, 23:26:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:26:57 DEBUG 23:26:57 DEBUG "value": [ 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG }, 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "0" 23:26:57 DEBUG 23:26:57 DEBUG "0" 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG ] 23:26:57 DEBUG [...] 23:26:57 DEBUG ] 23:26:57 DEBUG "instance": "10.142.0.43:9100" 23:26:57 DEBUG 23:26:57 DEBUG } 23:26:57 DEBUG 23:26:57 DEBUG } 23:26:57 DEBUG }, 23:26:57 DEBUG 23:26:57 DEBUG ] 23:26:57 DEBUG 23:26:57 DEBUG ] 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG } 23:26:57 DEBUG 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG } 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:26:57 DEBUG } 23:26:57 DEBUG "pod": "ds-cts-2" 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG } 23:26:57 DEBUG "12131.933333333332" 23:26:57 DEBUG }, 23:26:57 DEBUG }, 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG ] 23:26:57 DEBUG "value": [ 23:26:57 DEBUG "value": [ 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG } 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:26:57 DEBUG ] 23:26:57 DEBUG "2437636096" 23:26:57 DEBUG "0" 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG ] 23:26:57 DEBUG { 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG ] 23:26:57 DEBUG ] 23:26:57 DEBUG "data": { 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "4.185661374544768" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "instance": "10.142.0.107:9100" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "instance": "10.142.0.43:9100" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "13908210688" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220017" 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "ds-idrepo-2" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "ds-idrepo-0" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "0" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "instance": "10.142.0.107:9100" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "instance": "10.142.0.43:9100" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "0.0003333333333330302" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "ds-idrepo-2" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "ds-idrepo-0" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "0" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "0" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "instance": "10.142.0.107:9100" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "instance": "10.142.0.43:9100" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "4414.133333333333" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "2292.8" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "2293.0666666666666" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "0" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "instance": "10.142.0.107:9100" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "instance": "10.142.0.43:9100" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "0" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "ds-idrepo-2" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "ds-idrepo-0" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "0" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "0.082" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:26:57 DEBUG "pod": "ds-idrepo-2" 23:26:57 DEBUG }, 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:26:57 DEBUG "pod": "ds-idrepo-0" 23:26:57 DEBUG }, 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "412202.5" 23:26:57 DEBUG "0" 23:26:57 DEBUG ] 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220017" 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "25833.118024691357" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "6.745635762265577" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "2293.6" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "ds-idrepo-2" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "ds-idrepo-0" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "0" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220017" 23:26:57 INFO [http_cmd]: http status code OK 23:26:57 DEBUG --- status code --- 23:26:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:26:57 DEBUG --- http response --- 23:26:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:26:57 DEBUG ----- output ----- 23:26:57 DEBUG { 23:26:57 DEBUG "status": "success", 23:26:57 DEBUG "data": { 23:26:57 DEBUG "resultType": "vector", 23:26:57 DEBUG "result": [ 23:26:57 DEBUG { 23:26:57 DEBUG "metric": { 23:26:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG [...] 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG 23:26:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:26:57 DEBUG }, 23:26:57 DEBUG "value": [ 23:26:57 DEBUG 1713220017, 23:26:57 DEBUG "2297.8333333333335" 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG ] 23:26:57 DEBUG } 23:26:57 DEBUG } 23:26:58 INFO 23:26:58 INFO [loop_until]: kubectl --namespace=xlou top pods 23:26:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:26:58 INFO [loop_until]: OK (rc = 0) 23:26:58 DEBUG --- stdout --- 23:26:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13072m 8644Mi am-7d597cf7d6-8vr8f 10279m 8896Mi am-7d597cf7d6-9wz9h 15569m 9022Mi ds-cts-0 15926m 8563Mi ds-cts-1 10830m 8625Mi ds-cts-2 10829m 8780Mi ds-idrepo-0 13454m 4983Mi ds-idrepo-1 1774m 2746Mi ds-idrepo-2 1861m 2759Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6649m 2230Mi 23:26:58 DEBUG --- stderr --- 23:26:58 DEBUG 23:26:58 INFO 23:26:58 INFO [loop_until]: kubectl --namespace=xlou top node 23:26:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:26:58 INFO [loop_until]: OK (rc = 0) 23:26:58 DEBUG --- stdout --- 23:26:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 5069Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10705m 35% 10919Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13214m 44% 10724Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17590m 58% 3066Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1961m 6% 5653Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16568m 55% 12877Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2978Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10293m 34% 10406Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1904m 6% 5008Mi 4% gke-xlou-cdm-ds-0f601751-j488 11017m 36% 10440Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2440Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17373m 58% 10676Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13807m 46% 7159Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6656m 41% 6085Mi 10% 23:26:58 DEBUG --- stderr --- 23:26:58 DEBUG 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 ERROR [http_cmd]: ERROR 23:27:57 ERROR --- status code --- 23:27:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:27:57 ERROR --- http response --- 23:27:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG --- status code --- 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG { 23:27:57 DEBUG { 23:27:57 DEBUG { 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG { 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG "data": { 23:27:57 DEBUG "data": { 23:27:57 DEBUG "data": { 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:27:57 DEBUG "data": { 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG "result": [ 23:27:57 DEBUG "result": [ 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG { 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG { 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG { 23:27:57 DEBUG { 23:27:57 DEBUG { 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG { 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "data": { 23:27:57 DEBUG "data": { 23:27:57 DEBUG "data": { 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "data": { 23:27:57 DEBUG "data": { 23:27:57 DEBUG "pod": "ds-idrepo-2" 23:27:57 DEBUG "pod": "ds-idrepo-2" 23:27:57 DEBUG "instance": "10.142.0.107:9100" 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG "data": { 23:27:57 DEBUG "pod": "ds-idrepo-2" 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG }, 23:27:57 DEBUG }, 23:27:57 DEBUG }, 23:27:57 DEBUG "result": [ 23:27:57 DEBUG "result": [ 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG }, 23:27:57 DEBUG "result": [ 23:27:57 DEBUG "result": [ 23:27:57 DEBUG "value": [ 23:27:57 DEBUG "value": [ 23:27:57 DEBUG "value": [ 23:27:57 DEBUG { 23:27:57 DEBUG { 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "result": [ 23:27:57 DEBUG "value": [ 23:27:57 DEBUG { 23:27:57 DEBUG { 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "data": { 23:27:57 DEBUG { 23:27:57 DEBUG 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:27:57 DEBUG "instance": "10.142.0.107:9100" 23:27:57 DEBUG "instance": "10.142.0.107:9100" 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "metric": { 23:27:57 DEBUG 23:27:57 DEBUG "pod": "ds-idrepo-2" 23:27:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG }, 23:27:57 DEBUG }, 23:27:57 DEBUG }, 23:27:57 DEBUG "result": [ 23:27:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:27:57 DEBUG 23:27:57 DEBUG }, 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG [...] 23:27:57 DEBUG [...] 23:27:57 DEBUG "value": [ 23:27:57 DEBUG "value": [ 23:27:57 DEBUG "value": [ 23:27:57 DEBUG { 23:27:57 DEBUG }, 23:27:57 DEBUG [...] 23:27:57 DEBUG "value": [ 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "value": [ 23:27:57 DEBUG }, 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG "pod": "ds-idrepo-0" 23:27:57 DEBUG "instance": "10.142.0.43:9100" 23:27:57 DEBUG [...] 23:27:57 DEBUG [...] 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "pod": "ds-cts-2" 23:27:57 DEBUG [...] 23:27:57 DEBUG "pod": "ds-idrepo-0" 23:27:57 DEBUG 23:27:57 DEBUG }, 23:27:57 DEBUG }, 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG }, 23:27:57 DEBUG 23:27:57 DEBUG }, 23:27:57 DEBUG 23:27:57 DEBUG "value": [ 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:27:57 DEBUG "0" 23:27:57 DEBUG "0.00040000000000001514" 23:27:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:27:57 DEBUG "instance": "10.142.0.43:9100" 23:27:57 DEBUG "instance": "10.142.0.43:9100" 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "0" 23:27:57 DEBUG "pod": "ds-idrepo-0" 23:27:57 DEBUG "0" 23:27:57 DEBUG }, 23:27:57 DEBUG ] 23:27:57 DEBUG ] 23:27:57 DEBUG }, 23:27:57 DEBUG }, 23:27:57 DEBUG }, 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG ] 23:27:57 DEBUG }, 23:27:57 DEBUG ] 23:27:57 DEBUG "value": [ 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG "value": [ 23:27:57 DEBUG "value": [ 23:27:57 DEBUG "value": [ 23:27:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:27:57 DEBUG 23:27:57 DEBUG } 23:27:57 DEBUG "value": [ 23:27:57 DEBUG } 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG ] 23:27:57 DEBUG ] 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG }, 23:27:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:27:57 DEBUG ] 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG ] 23:27:57 DEBUG "0" 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG "0" 23:27:57 DEBUG "11913.499999999998" 23:27:57 DEBUG "13908859904" 23:27:57 DEBUG "value": [ 23:27:57 DEBUG }, 23:27:57 DEBUG } 23:27:57 DEBUG "0" 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG ] 23:27:57 DEBUG ] 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG "2436218880" 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG ] 23:27:57 DEBUG ] 23:27:57 DEBUG ] 23:27:57 DEBUG ] 23:27:57 DEBUG "0" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG { 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG "data": { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:27:57 DEBUG "result": [ 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG { 23:27:57 DEBUG { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "instance": "10.142.0.107:9100" 23:27:57 DEBUG "data": { 23:27:57 DEBUG }, 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "value": [ 23:27:57 DEBUG "result": [ 23:27:57 DEBUG 23:27:57 DEBUG { 23:27:57 DEBUG 23:27:57 DEBUG "metric": { 23:27:57 DEBUG 23:27:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:27:57 DEBUG [...] 23:27:57 DEBUG }, 23:27:57 DEBUG 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "instance": "10.142.0.43:9100" 23:27:57 DEBUG 23:27:57 DEBUG }, 23:27:57 DEBUG [...] 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG 23:27:57 DEBUG "2987.333333333333" 23:27:57 DEBUG 23:27:57 DEBUG ] 23:27:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:27:57 DEBUG } 23:27:57 DEBUG }, 23:27:57 DEBUG ] 23:27:57 DEBUG "value": [ 23:27:57 DEBUG } 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG } 23:27:57 DEBUG "2286.5333333333333" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220077" 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "data": { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "2286.5333333333333" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "data": { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "0" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "data": { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "pod": "ds-idrepo-2" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "pod": "ds-idrepo-0" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "0" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "data": { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "instance": "10.142.0.107:9100" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "instance": "10.142.0.43:9100" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "0.00033333333333333365" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "data": { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "pod": "ds-idrepo-2" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "pod": "ds-idrepo-0" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "0" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "data": { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "0.082" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "data": { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "413252" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220077" 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "data": { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "0" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "data": { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "6.707656317181506" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "data": { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "2291.233333333333" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "data": { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "pod": "ds-idrepo-2" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "pod": "ds-idrepo-0" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "0" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220077" 23:27:57 INFO [http_cmd]: http status code OK 23:27:57 DEBUG --- status code --- 23:27:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:27:57 DEBUG --- http response --- 23:27:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:27:57 DEBUG ----- output ----- 23:27:57 DEBUG { 23:27:57 DEBUG "status": "success", 23:27:57 DEBUG "data": { 23:27:57 DEBUG "resultType": "vector", 23:27:57 DEBUG "result": [ 23:27:57 DEBUG { 23:27:57 DEBUG "metric": { 23:27:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG [...] 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG 23:27:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:27:57 DEBUG }, 23:27:57 DEBUG "value": [ 23:27:57 DEBUG 1713220077, 23:27:57 DEBUG "2321.2666666666664" 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG ] 23:27:57 DEBUG } 23:27:57 DEBUG } 23:27:58 INFO 23:27:58 INFO [loop_until]: kubectl --namespace=xlou top pods 23:27:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:27:58 INFO [loop_until]: OK (rc = 0) 23:27:58 DEBUG --- stdout --- 23:27:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12927m 8645Mi am-7d597cf7d6-8vr8f 9968m 8894Mi am-7d597cf7d6-9wz9h 16343m 9015Mi ds-cts-0 15359m 8555Mi ds-cts-1 11330m 8635Mi ds-cts-2 10913m 8774Mi ds-idrepo-0 13635m 4987Mi ds-idrepo-1 1819m 2746Mi ds-idrepo-2 1838m 2759Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6596m 2231Mi 23:27:58 DEBUG --- stderr --- 23:27:58 DEBUG 23:27:58 INFO 23:27:58 INFO [loop_until]: kubectl --namespace=xlou top node 23:27:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:27:58 INFO [loop_until]: OK (rc = 0) 23:27:58 DEBUG --- stdout --- 23:27:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5073Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10989m 36% 10934Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13203m 44% 10722Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17559m 58% 3065Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1963m 6% 5637Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16751m 56% 12903Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2966Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10386m 34% 10400Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1945m 6% 5000Mi 4% gke-xlou-cdm-ds-0f601751-j488 10759m 36% 10442Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2443Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17248m 57% 10667Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13848m 46% 7171Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6725m 42% 6087Mi 10% 23:27:58 DEBUG --- stderr --- 23:27:58 DEBUG 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220137" 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "pod": "ds-idrepo-2" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "pod": "ds-idrepo-0" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "0" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "instance": "10.142.0.107:9100" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "instance": "10.142.0.43:9100" 23:28:57 DEBUG }, 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 ERROR [http_cmd]: ERROR 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG "value": [ 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 ERROR --- status code --- 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG "13909756928" 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 ERROR --- http response --- 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG ] 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG ----- output ----- 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG ] 23:28:57 DEBUG --- status code --- 23:28:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG { 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG } 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG { 23:28:57 DEBUG { 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG } 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG "data": { 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:28:57 DEBUG "data": { 23:28:57 DEBUG "data": { 23:28:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG "result": [ 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "result": [ 23:28:57 DEBUG "result": [ 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG { 23:28:57 DEBUG { 23:28:57 DEBUG { 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG { 23:28:57 DEBUG { 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "data": { 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220137" 23:28:57 DEBUG "pod": "ds-idrepo-2" 23:28:57 DEBUG "data": { 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "instance": "10.142.0.107:9100" 23:28:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG }, 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG }, 23:28:57 DEBUG }, 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "value": [ 23:28:57 DEBUG "result": [ 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "value": [ 23:28:57 DEBUG "value": [ 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG 23:28:57 DEBUG { 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "data": { 23:28:57 DEBUG 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "pod": "ds-idrepo-2" 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG 23:28:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:28:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:28:57 DEBUG }, 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG }, 23:28:57 DEBUG "result": [ 23:28:57 DEBUG [...] 23:28:57 DEBUG }, 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG [...] 23:28:57 DEBUG [...] 23:28:57 DEBUG "value": [ 23:28:57 DEBUG { 23:28:57 DEBUG 23:28:57 DEBUG "value": [ 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "metric": { 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "instance": "10.142.0.107:9100" 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG }, 23:28:57 DEBUG "pod": "ds-idrepo-0" 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG "instance": "10.142.0.43:9100" 23:28:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:28:57 DEBUG [...] 23:28:57 DEBUG "value": [ 23:28:57 DEBUG }, 23:28:57 DEBUG [...] 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG }, 23:28:57 DEBUG }, 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "value": [ 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "0" 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:28:57 DEBUG "0.00036666666666652263" 23:28:57 DEBUG "6.58442052846517" 23:28:57 DEBUG "pod": "ds-idrepo-0" 23:28:57 DEBUG [...] 23:28:57 DEBUG ] 23:28:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:28:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:28:57 DEBUG }, 23:28:57 DEBUG ] 23:28:57 DEBUG ] 23:28:57 DEBUG }, 23:28:57 DEBUG 23:28:57 DEBUG } 23:28:57 DEBUG }, 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG ] 23:28:57 DEBUG "value": [ 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG ] 23:28:57 DEBUG ] 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG } 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "0" 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 DEBUG "0" 23:28:57 DEBUG "instance": "10.142.0.43:9100" 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG } 23:28:57 DEBUG "0" 23:28:57 DEBUG "2286.6666666666665" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG }, 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG ] 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 DEBUG "value": [ 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG ] 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG ] 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 DEBUG "12279.033333333331" 23:28:57 DEBUG } 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG { 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG ] 23:28:57 DEBUG "data": { 23:28:57 DEBUG } 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG } 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "pod": "ds-idrepo-2" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "pod": "ds-cts-2" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "0" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "2437054464" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220137" 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "instance": "10.142.0.107:9100" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "instance": "10.142.0.43:9100" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "4832.533333333334" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "pod": "ds-idrepo-2" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "pod": "ds-idrepo-0" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "0" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220137" 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "2286.0666666666666" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "0" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "instance": "10.142.0.107:9100" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "instance": "10.142.0.43:9100" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "0" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:28:57 DEBUG 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG 23:28:57 DEBUG { 23:28:57 DEBUG 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:28:57 DEBUG "data": { 23:28:57 DEBUG }, 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "value": [ 23:28:57 DEBUG "result": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG { 23:28:57 DEBUG "0.082" 23:28:57 DEBUG "metric": { 23:28:57 DEBUG ] 23:28:57 DEBUG "pod": "ds-idrepo-2" 23:28:57 DEBUG } 23:28:57 DEBUG }, 23:28:57 DEBUG ] 23:28:57 DEBUG "value": [ 23:28:57 DEBUG } 23:28:57 DEBUG 23:28:57 DEBUG } 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "pod": "ds-idrepo-0" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "0" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "413845.5" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "62172.59314297845" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220137" 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "6.6186971902287866" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "2299.133333333333" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "pod": "ds-idrepo-2" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "pod": "ds-idrepo-0" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "0" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220137" 23:28:57 INFO [http_cmd]: http status code OK 23:28:57 DEBUG --- status code --- 23:28:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:28:57 DEBUG --- http response --- 23:28:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:28:57 DEBUG ----- output ----- 23:28:57 DEBUG { 23:28:57 DEBUG "status": "success", 23:28:57 DEBUG "data": { 23:28:57 DEBUG "resultType": "vector", 23:28:57 DEBUG "result": [ 23:28:57 DEBUG { 23:28:57 DEBUG "metric": { 23:28:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG [...] 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG 23:28:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:28:57 DEBUG }, 23:28:57 DEBUG "value": [ 23:28:57 DEBUG 1713220137, 23:28:57 DEBUG "2272.366666666667" 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG ] 23:28:57 DEBUG } 23:28:57 DEBUG } 23:28:58 INFO 23:28:58 INFO [loop_until]: kubectl --namespace=xlou top pods 23:28:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:28:58 INFO [loop_until]: OK (rc = 0) 23:28:58 DEBUG --- stdout --- 23:28:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12977m 8640Mi am-7d597cf7d6-8vr8f 10268m 8917Mi am-7d597cf7d6-9wz9h 15943m 8991Mi ds-cts-0 15167m 8558Mi ds-cts-1 10864m 8653Mi ds-cts-2 10841m 8778Mi ds-idrepo-0 13655m 4987Mi ds-idrepo-1 1841m 2746Mi ds-idrepo-2 1847m 2758Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6582m 2233Mi 23:28:58 DEBUG --- stderr --- 23:28:58 DEBUG 23:28:58 INFO 23:28:58 INFO [loop_until]: kubectl --namespace=xlou top node 23:28:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:28:58 INFO [loop_until]: OK (rc = 0) 23:28:58 DEBUG --- stdout --- 23:28:58 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5073Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10812m 36% 10937Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13082m 43% 10719Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17528m 58% 3062Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1976m 6% 5644Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16196m 54% 12892Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 55m 0% 2968Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10225m 34% 10424Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1931m 6% 5011Mi 4% gke-xlou-cdm-ds-0f601751-j488 10384m 34% 10457Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2461Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17214m 57% 10652Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13846m 46% 7164Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6763m 42% 6087Mi 10% 23:28:58 DEBUG --- stderr --- 23:28:58 DEBUG 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 ERROR [http_cmd]: ERROR 23:29:57 ERROR --- status code --- 23:29:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:29:57 ERROR --- http response --- 23:29:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:29:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "pod": "ds-idrepo-2" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "pod": "ds-idrepo-0" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "0" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG { 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "instance": "10.142.0.107:9100" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "instance": "10.142.0.43:9100" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "12973.233333333332" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG } 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG { 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG "data": { 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "data": { 23:29:57 DEBUG "pod": "ds-idrepo-2" 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG }, 23:29:57 DEBUG "result": [ 23:29:57 DEBUG "value": [ 23:29:57 DEBUG { 23:29:57 DEBUG 23:29:57 DEBUG "metric": { 23:29:57 DEBUG 23:29:57 DEBUG "instance": "10.142.0.107:9100" 23:29:57 DEBUG 23:29:57 DEBUG }, 23:29:57 DEBUG [...] 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG "pod": "ds-cts-2" 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG [...] 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG }, 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG "value": [ 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG "0" 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:29:57 DEBUG "instance": "10.142.0.43:9100" 23:29:57 DEBUG { 23:29:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:29:57 DEBUG ] 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG }, 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG } 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:29:57 DEBUG { 23:29:57 DEBUG "value": [ 23:29:57 DEBUG "data": { 23:29:57 DEBUG ] 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG } 23:29:57 DEBUG { 23:29:57 DEBUG "data": { 23:29:57 DEBUG "0.00046666666666700016" 23:29:57 DEBUG "result": [ 23:29:57 DEBUG } 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG ] 23:29:57 DEBUG { 23:29:57 DEBUG "data": { 23:29:57 DEBUG "result": [ 23:29:57 DEBUG } 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG { 23:29:57 DEBUG ] 23:29:57 DEBUG "instance": "10.142.0.107:9100" 23:29:57 DEBUG "result": [ 23:29:57 DEBUG "metric": { 23:29:57 DEBUG } 23:29:57 DEBUG }, 23:29:57 DEBUG { 23:29:57 DEBUG "pod": "ds-cts-1" 23:29:57 DEBUG } 23:29:57 DEBUG "value": [ 23:29:57 DEBUG "metric": { 23:29:57 DEBUG }, 23:29:57 DEBUG 23:29:57 DEBUG "pod": "ds-cts-1" 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG }, 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG "instance": "10.142.0.43:9100" 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG }, 23:29:57 DEBUG 23:29:57 DEBUG "pod": "ds-idrepo-2" 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG }, 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "pod": "ds-idrepo-2" 23:29:57 DEBUG "value": [ 23:29:57 DEBUG "13910305792" 23:29:57 DEBUG }, 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG ] 23:29:57 DEBUG "value": [ 23:29:57 DEBUG "110.34505148886522" 23:29:57 DEBUG } 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG ] 23:29:57 DEBUG ] 23:29:57 DEBUG "3531.041647643687" 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "pod": "ds-idrepo-2" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "pod": "ds-idrepo-0" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "0" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220197" 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "pod": "ds-idrepo-2" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "pod": "ds-idrepo-0" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "0" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "instance": "10.142.0.107:9100" 23:29:57 DEBUG "pod": "ds-cts-1" 23:29:57 DEBUG }, 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "instance": "10.142.0.43:9100" 23:29:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:29:57 DEBUG }, 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "3334.5" 23:29:57 DEBUG "0" 23:29:57 DEBUG ] 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG { 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "data": { 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG "result": [ 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG { 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "pod": "ds-idrepo-2" 23:29:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:29:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:29:57 DEBUG }, 23:29:57 DEBUG }, 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG "value": [ 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG [...] 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "pod": "ds-idrepo-0" 23:29:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:29:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:29:57 DEBUG }, 23:29:57 DEBUG }, 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG "value": [ 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "0" 23:29:57 DEBUG "2438103040" 23:29:57 DEBUG "2300.0666666666666" 23:29:57 DEBUG ] 23:29:57 DEBUG ] 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG ] 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "pod": "ds-idrepo-2" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "pod": "ds-idrepo-0" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "0" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220197" 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:29:57 DEBUG 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG [...] 23:29:57 DEBUG { 23:29:57 DEBUG 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG 23:29:57 DEBUG "data": { 23:29:57 DEBUG 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:29:57 DEBUG "result": [ 23:29:57 DEBUG }, 23:29:57 DEBUG { 23:29:57 DEBUG "value": [ 23:29:57 DEBUG "metric": { 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:29:57 DEBUG "2299.133333333333" 23:29:57 DEBUG }, 23:29:57 DEBUG ] 23:29:57 DEBUG "value": [ 23:29:57 DEBUG } 23:29:57 DEBUG 23:29:57 DEBUG ] 23:29:57 DEBUG 23:29:57 DEBUG } 23:29:57 DEBUG 23:29:57 DEBUG } 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "0" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "instance": "10.142.0.107:9100" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "instance": "10.142.0.43:9100" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "0.00033333333333333365" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "0.082" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "414162" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "pod": "ds-cts-1" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "0" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220197" 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "6.60201073959283" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "2298.866666666667" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "pod": "ds-idrepo-2" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "pod": "ds-idrepo-0" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "0" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220197" 23:29:57 INFO [http_cmd]: http status code OK 23:29:57 DEBUG --- status code --- 23:29:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:29:57 DEBUG --- http response --- 23:29:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:29:57 DEBUG ----- output ----- 23:29:57 DEBUG { 23:29:57 DEBUG "status": "success", 23:29:57 DEBUG "data": { 23:29:57 DEBUG "resultType": "vector", 23:29:57 DEBUG "result": [ 23:29:57 DEBUG { 23:29:57 DEBUG "metric": { 23:29:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG [...] 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG 23:29:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:29:57 DEBUG }, 23:29:57 DEBUG "value": [ 23:29:57 DEBUG 1713220197, 23:29:57 DEBUG "2288.3333333333335" 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG ] 23:29:57 DEBUG } 23:29:57 DEBUG } 23:29:58 INFO 23:29:58 INFO [loop_until]: kubectl --namespace=xlou top pods 23:29:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:29:58 INFO [loop_until]: OK (rc = 0) 23:29:58 DEBUG --- stdout --- 23:29:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13052m 8663Mi am-7d597cf7d6-8vr8f 10091m 8912Mi am-7d597cf7d6-9wz9h 16117m 8998Mi ds-cts-0 15408m 8586Mi ds-cts-1 11004m 8648Mi ds-cts-2 10610m 8780Mi ds-idrepo-0 13521m 4988Mi ds-idrepo-1 1853m 2747Mi ds-idrepo-2 1800m 2759Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 4m 1224Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6719m 2234Mi 23:29:58 DEBUG --- stderr --- 23:29:58 DEBUG 23:29:58 INFO 23:29:58 INFO [loop_until]: kubectl --namespace=xlou top node 23:29:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:29:59 INFO [loop_until]: OK (rc = 0) 23:29:59 DEBUG --- stdout --- 23:29:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5074Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10889m 36% 10934Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13159m 44% 10736Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17589m 58% 3059Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1967m 6% 5641Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16200m 54% 12903Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2968Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 9917m 33% 10412Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1901m 6% 4993Mi 4% gke-xlou-cdm-ds-0f601751-j488 10854m 36% 10461Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2440Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17234m 57% 10649Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13890m 46% 7164Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6759m 42% 6090Mi 10% 23:29:59 DEBUG --- stderr --- 23:29:59 DEBUG 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220257" 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 ERROR [http_cmd]: ERROR 23:30:57 ERROR --- status code --- 23:30:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:30:57 ERROR --- http response --- 23:30:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:30:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "0" 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220257" 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220257" 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220257" 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220257" 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG --- http response --- 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG { 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "data": { 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG "result": [ 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG { 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "instance": "10.142.0.107:9100" 23:30:57 DEBUG "pod": "ds-idrepo-2" 23:30:57 DEBUG "instance": "10.142.0.107:9100" 23:30:57 DEBUG }, 23:30:57 DEBUG }, 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG "value": [ 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG [...] 23:30:57 DEBUG [...] 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG [...] 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:30:57 DEBUG 23:30:57 DEBUG "instance": "10.142.0.43:9100" 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG "pod": "ds-idrepo-0" 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG "instance": "10.142.0.43:9100" 23:30:57 DEBUG }, 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG }, 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG { 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG { 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG "value": [ 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:30:57 DEBUG "data": { 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "3234.7" 23:30:57 DEBUG "data": { 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG "0" 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG "12378.999999999998" 23:30:57 DEBUG ] 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG { 23:30:57 DEBUG ] 23:30:57 DEBUG { 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG "result": [ 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG } 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG { 23:30:57 DEBUG "data": { 23:30:57 DEBUG ] 23:30:57 DEBUG "data": { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "data": { 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG } 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "instance": "10.142.0.107:9100" 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 DEBUG "instance": "10.142.0.107:9100" 23:30:57 DEBUG "result": [ 23:30:57 DEBUG } 23:30:57 DEBUG "result": [ 23:30:57 DEBUG }, 23:30:57 DEBUG "result": [ 23:30:57 DEBUG } 23:30:57 DEBUG }, 23:30:57 DEBUG { 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG { 23:30:57 DEBUG "value": [ 23:30:57 DEBUG { 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 DEBUG "value": [ 23:30:57 DEBUG "metric": { 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG "metric": { 23:30:57 DEBUG 23:30:57 DEBUG "metric": { 23:30:57 DEBUG 23:30:57 DEBUG "pod": "ds-idrepo-2" 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG "pod": "ds-idrepo-2" 23:30:57 DEBUG 23:30:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:30:57 DEBUG 23:30:57 DEBUG }, 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG }, 23:30:57 DEBUG 23:30:57 DEBUG }, 23:30:57 DEBUG 23:30:57 DEBUG "value": [ 23:30:57 DEBUG "value": [ 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:30:57 DEBUG [...] 23:30:57 DEBUG "value": [ 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG { 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG [...] 23:30:57 DEBUG "data": { 23:30:57 DEBUG "instance": "10.142.0.43:9100" 23:30:57 DEBUG [...] 23:30:57 DEBUG "instance": "10.142.0.43:9100" 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG }, 23:30:57 DEBUG 23:30:57 DEBUG }, 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "result": [ 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG { 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "pod": "ds-cts-2" 23:30:57 DEBUG "pod": "ds-idrepo-0" 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "13910936576" 23:30:57 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:30:57 DEBUG "0.00036666666666652263" 23:30:57 DEBUG }, 23:30:57 DEBUG }, 23:30:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:30:57 DEBUG ] 23:30:57 DEBUG }, 23:30:57 DEBUG ] 23:30:57 DEBUG "value": [ 23:30:57 DEBUG "value": [ 23:30:57 DEBUG }, 23:30:57 DEBUG } 23:30:57 DEBUG "value": [ 23:30:57 DEBUG } 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG ] 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG ] 23:30:57 DEBUG "0" 23:30:57 DEBUG "0" 23:30:57 DEBUG 23:30:57 DEBUG } 23:30:57 DEBUG "0" 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG ] 23:30:57 DEBUG 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG [...] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 DEBUG 23:30:57 DEBUG } 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "5.385220703059053" 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220257" 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "pod": "ds-idrepo-2" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "pod": "ds-idrepo-0" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "0" 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "pod": "ds-idrepo-2" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "pod": "ds-idrepo-0" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "0" 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "2301.3" 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "2440269824" 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220257" 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "instance": "10.142.0.107:9100" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "instance": "10.142.0.43:9100" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "0" 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "pod": "ds-idrepo-2" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "pod": "ds-idrepo-0" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "0" 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "2302.6" 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "0" 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG { 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:30:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:30:57 DEBUG }, 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:30:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:30:57 DEBUG }, 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "413320.5" 23:30:57 DEBUG "0.082" 23:30:57 DEBUG ] 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "303634179.8351704" 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220257" 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "6.651183199494437" 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220257" 23:30:57 INFO [http_cmd]: http status code OK 23:30:57 DEBUG --- status code --- 23:30:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:57 DEBUG --- http response --- 23:30:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:30:57 DEBUG ----- output ----- 23:30:57 DEBUG { 23:30:57 DEBUG "status": "success", 23:30:57 DEBUG "data": { 23:30:57 DEBUG "resultType": "vector", 23:30:57 DEBUG "result": [ 23:30:57 DEBUG { 23:30:57 DEBUG "metric": { 23:30:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG [...] 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG 23:30:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:30:57 DEBUG }, 23:30:57 DEBUG "value": [ 23:30:57 DEBUG 1713220257, 23:30:57 DEBUG "2294" 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG ] 23:30:57 DEBUG } 23:30:57 DEBUG } 23:30:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220258" 23:30:58 INFO [http_cmd]: http status code OK 23:30:58 DEBUG --- status code --- 23:30:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:58 DEBUG --- http response --- 23:30:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:30:58 DEBUG ----- output ----- 23:30:58 DEBUG { 23:30:58 DEBUG "status": "success", 23:30:58 DEBUG "data": { 23:30:58 DEBUG "resultType": "vector", 23:30:58 DEBUG "result": [ 23:30:58 DEBUG { 23:30:58 DEBUG "metric": { 23:30:58 DEBUG "pod": "ds-idrepo-2" 23:30:58 DEBUG }, 23:30:58 DEBUG "value": [ 23:30:58 DEBUG 23:30:58 DEBUG 23:30:58 DEBUG 23:30:58 DEBUG [...] 23:30:58 DEBUG 23:30:58 DEBUG 23:30:58 DEBUG 23:30:58 DEBUG "pod": "ds-idrepo-0" 23:30:58 DEBUG }, 23:30:58 DEBUG "value": [ 23:30:58 DEBUG 1713220258, 23:30:58 DEBUG "0" 23:30:58 DEBUG ] 23:30:58 DEBUG } 23:30:58 DEBUG ] 23:30:58 DEBUG } 23:30:58 DEBUG } 23:30:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220258" 23:30:58 INFO [http_cmd]: http status code OK 23:30:58 DEBUG --- status code --- 23:30:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:30:58 DEBUG --- http response --- 23:30:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:30:58 DEBUG ----- output ----- 23:30:58 DEBUG { 23:30:58 DEBUG "status": "success", 23:30:58 DEBUG "data": { 23:30:58 DEBUG "resultType": "vector", 23:30:58 DEBUG "result": [ 23:30:58 DEBUG { 23:30:58 DEBUG "metric": { 23:30:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:30:58 DEBUG }, 23:30:58 DEBUG "value": [ 23:30:58 DEBUG 23:30:58 DEBUG 23:30:58 DEBUG 23:30:58 DEBUG [...] 23:30:58 DEBUG 23:30:58 DEBUG 23:30:58 DEBUG 23:30:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:30:58 DEBUG }, 23:30:58 DEBUG "value": [ 23:30:58 DEBUG 1713220258, 23:30:58 DEBUG "2288.5" 23:30:58 DEBUG ] 23:30:58 DEBUG } 23:30:58 DEBUG ] 23:30:58 DEBUG } 23:30:58 DEBUG } 23:30:58 INFO 23:30:58 INFO [loop_until]: kubectl --namespace=xlou top pods 23:30:58 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:30:58 INFO [loop_until]: OK (rc = 0) 23:30:58 DEBUG --- stdout --- 23:30:58 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12480m 8639Mi am-7d597cf7d6-8vr8f 9992m 8910Mi am-7d597cf7d6-9wz9h 15746m 9000Mi ds-cts-0 15180m 8570Mi ds-cts-1 11179m 8661Mi ds-cts-2 10707m 8772Mi ds-idrepo-0 13557m 4988Mi ds-idrepo-1 1847m 2746Mi ds-idrepo-2 1890m 2760Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6550m 2236Mi 23:30:58 DEBUG --- stderr --- 23:30:58 DEBUG 23:30:59 INFO 23:30:59 INFO [loop_until]: kubectl --namespace=xlou top node 23:30:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:30:59 INFO [loop_until]: OK (rc = 0) 23:30:59 DEBUG --- stdout --- 23:30:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5074Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11062m 37% 10946Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13058m 43% 10722Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17532m 58% 3057Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1988m 6% 5641Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16266m 54% 12919Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2972Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10291m 34% 10421Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1915m 6% 4996Mi 4% gke-xlou-cdm-ds-0f601751-j488 10754m 36% 10453Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2441Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17242m 57% 10645Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13800m 46% 7162Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6870m 43% 6089Mi 10% 23:30:59 DEBUG --- stderr --- 23:30:59 DEBUG 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220317" 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220317" 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 ERROR [http_cmd]: ERROR 23:31:57 ERROR --- status code --- 23:31:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:31:57 ERROR --- http response --- 23:31:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:31:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "0" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG { 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "ds-idrepo-2" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "ds-idrepo-0" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "0" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "instance": "10.142.0.107:9100" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "instance": "10.142.0.43:9100" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "4537.366666666666" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "ds-idrepo-2" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "ds-idrepo-0" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "0" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "0" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "ds-idrepo-2" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "ds-cts-2" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "0" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220317" 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220317" 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220317" 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "ds-idrepo-2" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "ds-idrepo-0" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "0" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "instance": "10.142.0.107:9100" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "instance": "10.142.0.43:9100" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "12306.299999999997" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "instance": "10.142.0.107:9100" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:31:57 DEBUG 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG 23:31:57 DEBUG { 23:31:57 DEBUG 23:31:57 DEBUG "status": "success", 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG "instance": "10.142.0.43:9100" 23:31:57 DEBUG "data": { 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG }, 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG "value": [ 23:31:57 DEBUG "result": [ 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG { 23:31:57 DEBUG "0.00040000000000001514" 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:31:57 DEBUG "metric": { 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 DEBUG ] 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG "instance": "10.142.0.107:9100" 23:31:57 DEBUG } 23:31:57 DEBUG { 23:31:57 DEBUG }, 23:31:57 DEBUG ] 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "value": [ 23:31:57 DEBUG } 23:31:57 DEBUG "data": { 23:31:57 DEBUG 23:31:57 DEBUG } 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG 23:31:57 DEBUG "result": [ 23:31:57 DEBUG 23:31:57 DEBUG { 23:31:57 DEBUG [...] 23:31:57 DEBUG "metric": { 23:31:57 DEBUG 23:31:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:31:57 DEBUG 23:31:57 DEBUG }, 23:31:57 DEBUG 23:31:57 DEBUG "value": [ 23:31:57 DEBUG "instance": "10.142.0.43:9100" 23:31:57 DEBUG 23:31:57 DEBUG }, 23:31:57 DEBUG 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG [...] 23:31:57 DEBUG "13911583744" 23:31:57 DEBUG 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220317" 23:31:57 DEBUG ] 23:31:57 DEBUG 23:31:57 DEBUG } 23:31:57 DEBUG 23:31:57 DEBUG ] 23:31:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:31:57 DEBUG } 23:31:57 DEBUG }, 23:31:57 DEBUG } 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "0" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "ds-idrepo-2" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "ds-idrepo-0" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "0" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "2294.7666666666664" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "2442244096" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220317" 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "instance": "10.142.0.107:9100" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "instance": "10.142.0.43:9100" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "0.00033333333333333365" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "ds-idrepo-2" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "ds-idrepo-0" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "0" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "412699" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "2296.1" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "0" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220317" 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "0.082" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:57 INFO [http_cmd]: http status code OK 23:31:57 DEBUG --- status code --- 23:31:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:57 DEBUG --- http response --- 23:31:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:31:57 DEBUG ----- output ----- 23:31:57 DEBUG { 23:31:57 DEBUG "status": "success", 23:31:57 DEBUG "data": { 23:31:57 DEBUG "resultType": "vector", 23:31:57 DEBUG "result": [ 23:31:57 DEBUG { 23:31:57 DEBUG "metric": { 23:31:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG [...] 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG 23:31:57 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:31:57 DEBUG }, 23:31:57 DEBUG "value": [ 23:31:57 DEBUG 1713220317, 23:31:57 DEBUG "0" 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG ] 23:31:57 DEBUG } 23:31:57 DEBUG } 23:31:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220318" 23:31:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220318" 23:31:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220318" 23:31:58 INFO [http_cmd]: http status code OK 23:31:58 DEBUG --- status code --- 23:31:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:58 DEBUG --- http response --- 23:31:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:31:58 DEBUG ----- output ----- 23:31:58 DEBUG { 23:31:58 DEBUG "status": "success", 23:31:58 DEBUG "data": { 23:31:58 DEBUG "resultType": "vector", 23:31:58 DEBUG "result": [ 23:31:58 DEBUG { 23:31:58 DEBUG "metric": { 23:31:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:31:58 DEBUG }, 23:31:58 DEBUG "value": [ 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG [...] 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:31:58 DEBUG }, 23:31:58 DEBUG "value": [ 23:31:58 DEBUG 1713220318, 23:31:58 DEBUG "6.698839002485086" 23:31:58 DEBUG ] 23:31:58 DEBUG } 23:31:58 DEBUG ] 23:31:58 DEBUG } 23:31:58 DEBUG } 23:31:58 INFO [http_cmd]: http status code OK 23:31:58 DEBUG --- status code --- 23:31:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:58 DEBUG --- http response --- 23:31:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:31:58 DEBUG ----- output ----- 23:31:58 DEBUG { 23:31:58 DEBUG "status": "success", 23:31:58 DEBUG "data": { 23:31:58 DEBUG "resultType": "vector", 23:31:58 DEBUG "result": [ 23:31:58 DEBUG { 23:31:58 DEBUG "metric": { 23:31:58 DEBUG "pod": "ds-idrepo-2" 23:31:58 DEBUG }, 23:31:58 DEBUG "value": [ 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG [...] 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG "pod": "ds-idrepo-0" 23:31:58 DEBUG }, 23:31:58 DEBUG "value": [ 23:31:58 DEBUG 1713220318, 23:31:58 DEBUG "0" 23:31:58 DEBUG ] 23:31:58 DEBUG } 23:31:58 DEBUG ] 23:31:58 DEBUG } 23:31:58 DEBUG } 23:31:58 INFO [http_cmd]: http status code OK 23:31:58 DEBUG --- status code --- 23:31:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:58 DEBUG --- http response --- 23:31:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:31:58 DEBUG ----- output ----- 23:31:58 DEBUG { 23:31:58 DEBUG "status": "success", 23:31:58 DEBUG "data": { 23:31:58 DEBUG "resultType": "vector", 23:31:58 DEBUG "result": [ 23:31:58 DEBUG { 23:31:58 DEBUG "metric": { 23:31:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:31:58 DEBUG }, 23:31:58 DEBUG "value": [ 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG [...] 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:31:58 DEBUG }, 23:31:58 DEBUG "value": [ 23:31:58 DEBUG 1713220318, 23:31:58 DEBUG "2292.8333333333335" 23:31:58 DEBUG ] 23:31:58 DEBUG } 23:31:58 DEBUG ] 23:31:58 DEBUG } 23:31:58 DEBUG } 23:31:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220318" 23:31:58 INFO [http_cmd]: http status code OK 23:31:58 DEBUG --- status code --- 23:31:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:31:58 DEBUG --- http response --- 23:31:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:31:58 DEBUG ----- output ----- 23:31:58 DEBUG { 23:31:58 DEBUG "status": "success", 23:31:58 DEBUG "data": { 23:31:58 DEBUG "resultType": "vector", 23:31:58 DEBUG "result": [ 23:31:58 DEBUG { 23:31:58 DEBUG "metric": { 23:31:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:31:58 DEBUG }, 23:31:58 DEBUG "value": [ 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG [...] 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG 23:31:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:31:58 DEBUG }, 23:31:58 DEBUG "value": [ 23:31:58 DEBUG 1713220318, 23:31:58 DEBUG "2289.7666666666664" 23:31:58 DEBUG ] 23:31:58 DEBUG } 23:31:58 DEBUG ] 23:31:58 DEBUG } 23:31:58 DEBUG } 23:31:59 INFO 23:31:59 INFO [loop_until]: kubectl --namespace=xlou top pods 23:31:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:31:59 INFO [loop_until]: OK (rc = 0) 23:31:59 DEBUG --- stdout --- 23:31:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12632m 8637Mi am-7d597cf7d6-8vr8f 10428m 8911Mi am-7d597cf7d6-9wz9h 15906m 8992Mi ds-cts-0 15125m 8578Mi ds-cts-1 10844m 8637Mi ds-cts-2 10816m 8786Mi ds-idrepo-0 13290m 4988Mi ds-idrepo-1 1858m 2755Mi ds-idrepo-2 1839m 2763Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6758m 2239Mi 23:31:59 DEBUG --- stderr --- 23:31:59 DEBUG 23:31:59 INFO 23:31:59 INFO [loop_until]: kubectl --namespace=xlou top node 23:31:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:31:59 INFO [loop_until]: OK (rc = 0) 23:31:59 DEBUG --- stdout --- 23:31:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5095Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10920m 36% 10945Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13056m 43% 10718Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17579m 58% 3054Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1984m 6% 5637Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 15994m 53% 12892Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2963Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10301m 34% 10420Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1902m 6% 5001Mi 4% gke-xlou-cdm-ds-0f601751-j488 10810m 36% 10462Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2453Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17328m 58% 10653Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13789m 46% 7164Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6954m 43% 6089Mi 10% 23:31:59 DEBUG --- stderr --- 23:31:59 DEBUG 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220377" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220377" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 ERROR [http_cmd]: ERROR 23:32:57 ERROR --- status code --- 23:32:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:32:57 ERROR --- http response --- 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "pod": "ds-cts-1" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "3.287395398250904" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "instance": "10.142.0.107:9100" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "instance": "10.142.0.43:9100" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 DEBUG "3200.3666666666663" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 DEBUG ] 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG ] 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG } 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG } 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG { 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:32:57 DEBUG "result": [ 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "pod": "ds-idrepo-2" 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "pod": "ds-idrepo-2" 23:32:57 DEBUG "data": { 23:32:57 DEBUG }, 23:32:57 DEBUG "pod": "ds-cts-1" 23:32:57 DEBUG }, 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "value": [ 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG "result": [ 23:32:57 DEBUG 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG { 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "metric": { 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "pod": "ds-idrepo-2" 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG }, 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "pod": "ds-idrepo-0" 23:32:57 DEBUG 23:32:57 DEBUG "pod": "ds-idrepo-0" 23:32:57 DEBUG 23:32:57 DEBUG }, 23:32:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:32:57 DEBUG }, 23:32:57 DEBUG [...] 23:32:57 DEBUG "value": [ 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG 23:32:57 DEBUG "0" 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "0" 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG 23:32:57 DEBUG ] 23:32:57 DEBUG "0" 23:32:57 DEBUG ] 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG "pod": "ds-cts-2" 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:32:57 DEBUG }, 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG "value": [ 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 DEBUG "0" 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG } 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG } 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "pod": "ds-cts-1" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "0" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "pod": "ds-idrepo-2" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "pod": "ds-idrepo-0" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "0" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220377" 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "instance": "10.142.0.107:9100" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "instance": "10.142.0.43:9100" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "12397.899999999998" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "pod": "ds-idrepo-2" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "pod": "ds-idrepo-0" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "0" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220377" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220377" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220377" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220377" 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "instance": "10.142.0.107:9100" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "instance": "10.142.0.43:9100" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "0.00040000000000001514" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "instance": "10.142.0.107:9100" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "instance": "10.142.0.43:9100" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "13912249344" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "2289.6666666666665" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "instance": "10.142.0.107:9100" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "instance": "10.142.0.43:9100" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "0" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "2444443648" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "pod": "ds-idrepo-2" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "pod": "ds-idrepo-0" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "0" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "412731" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "2290.633333333333" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "0" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "pod": "ds-cts-1" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "63143010.52552734" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220377" 23:32:57 INFO [http_cmd]: http status code OK 23:32:57 DEBUG --- status code --- 23:32:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:57 DEBUG --- http response --- 23:32:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:32:57 DEBUG ----- output ----- 23:32:57 DEBUG { 23:32:57 DEBUG "status": "success", 23:32:57 DEBUG "data": { 23:32:57 DEBUG "resultType": "vector", 23:32:57 DEBUG "result": [ 23:32:57 DEBUG { 23:32:57 DEBUG "metric": { 23:32:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG [...] 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG 23:32:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:32:57 DEBUG }, 23:32:57 DEBUG "value": [ 23:32:57 DEBUG 1713220377, 23:32:57 DEBUG "0.082" 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG ] 23:32:57 DEBUG } 23:32:57 DEBUG } 23:32:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220378" 23:32:58 INFO [http_cmd]: http status code OK 23:32:58 DEBUG --- status code --- 23:32:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:58 DEBUG --- http response --- 23:32:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:32:58 DEBUG ----- output ----- 23:32:58 DEBUG { 23:32:58 DEBUG "status": "success", 23:32:58 DEBUG "data": { 23:32:58 DEBUG "resultType": "vector", 23:32:58 DEBUG "result": [ 23:32:58 DEBUG { 23:32:58 DEBUG "metric": { 23:32:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:32:58 DEBUG }, 23:32:58 DEBUG "value": [ 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG [...] 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:32:58 DEBUG }, 23:32:58 DEBUG "value": [ 23:32:58 DEBUG 1713220378, 23:32:58 DEBUG "6.743659583944027" 23:32:58 DEBUG ] 23:32:58 DEBUG } 23:32:58 DEBUG ] 23:32:58 DEBUG } 23:32:58 DEBUG } 23:32:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220378" 23:32:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220378" 23:32:58 INFO [http_cmd]: http status code OK 23:32:58 DEBUG --- status code --- 23:32:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:58 DEBUG --- http response --- 23:32:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:32:58 DEBUG ----- output ----- 23:32:58 DEBUG { 23:32:58 DEBUG "status": "success", 23:32:58 DEBUG "data": { 23:32:58 DEBUG "resultType": "vector", 23:32:58 DEBUG "result": [ 23:32:58 DEBUG { 23:32:58 DEBUG "metric": { 23:32:58 DEBUG "pod": "ds-idrepo-2" 23:32:58 DEBUG }, 23:32:58 DEBUG "value": [ 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG [...] 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG "pod": "ds-idrepo-0" 23:32:58 DEBUG }, 23:32:58 DEBUG "value": [ 23:32:58 DEBUG 1713220378, 23:32:58 DEBUG "0" 23:32:58 DEBUG ] 23:32:58 DEBUG } 23:32:58 DEBUG ] 23:32:58 DEBUG } 23:32:58 DEBUG } 23:32:58 INFO [http_cmd]: http status code OK 23:32:58 DEBUG --- status code --- 23:32:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:58 DEBUG --- http response --- 23:32:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:32:58 DEBUG ----- output ----- 23:32:58 DEBUG { 23:32:58 DEBUG "status": "success", 23:32:58 DEBUG "data": { 23:32:58 DEBUG "resultType": "vector", 23:32:58 DEBUG "result": [ 23:32:58 DEBUG { 23:32:58 DEBUG "metric": { 23:32:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:32:58 DEBUG }, 23:32:58 DEBUG "value": [ 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG [...] 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:32:58 DEBUG }, 23:32:58 DEBUG "value": [ 23:32:58 DEBUG 1713220378, 23:32:58 DEBUG "2293.9333333333334" 23:32:58 DEBUG ] 23:32:58 DEBUG } 23:32:58 DEBUG ] 23:32:58 DEBUG } 23:32:58 DEBUG } 23:32:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220378" 23:32:58 INFO [http_cmd]: http status code OK 23:32:58 DEBUG --- status code --- 23:32:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:32:58 DEBUG --- http response --- 23:32:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:32:58 DEBUG ----- output ----- 23:32:58 DEBUG { 23:32:58 DEBUG "status": "success", 23:32:58 DEBUG "data": { 23:32:58 DEBUG "resultType": "vector", 23:32:58 DEBUG "result": [ 23:32:58 DEBUG { 23:32:58 DEBUG "metric": { 23:32:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:32:58 DEBUG }, 23:32:58 DEBUG "value": [ 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG [...] 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG 23:32:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:32:58 DEBUG }, 23:32:58 DEBUG "value": [ 23:32:58 DEBUG 1713220378, 23:32:58 DEBUG "2297.7" 23:32:58 DEBUG ] 23:32:58 DEBUG } 23:32:58 DEBUG ] 23:32:58 DEBUG } 23:32:58 DEBUG } 23:32:59 INFO 23:32:59 INFO [loop_until]: kubectl --namespace=xlou top pods 23:32:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:32:59 INFO [loop_until]: OK (rc = 0) 23:32:59 DEBUG --- stdout --- 23:32:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12801m 8663Mi am-7d597cf7d6-8vr8f 10130m 8910Mi am-7d597cf7d6-9wz9h 15787m 9018Mi ds-cts-0 15437m 8564Mi ds-cts-1 11111m 8639Mi ds-cts-2 11179m 8786Mi ds-idrepo-0 13335m 4989Mi ds-idrepo-1 1824m 2755Mi ds-idrepo-2 1786m 2763Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6715m 2239Mi 23:32:59 DEBUG --- stderr --- 23:32:59 DEBUG 23:32:59 INFO 23:32:59 INFO [loop_until]: kubectl --namespace=xlou top node 23:32:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:32:59 INFO [loop_until]: OK (rc = 0) 23:32:59 DEBUG --- stdout --- 23:32:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5084Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11169m 37% 10933Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13194m 44% 10735Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17579m 58% 3062Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1918m 6% 5647Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16577m 55% 12909Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2965Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10259m 34% 10416Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1909m 6% 4999Mi 4% gke-xlou-cdm-ds-0f601751-j488 10685m 35% 10462Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2457Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16699m 55% 10677Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13790m 46% 7161Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6895m 43% 6090Mi 10% 23:32:59 DEBUG --- stderr --- 23:32:59 DEBUG 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220437" 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220437" 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220437" 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "result": [ 23:33:57 DEBUG { 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220437" 23:33:57 DEBUG "metric": { 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220437" 23:33:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG [...] 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG "10.055101637509642" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220437" 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "result": [ 23:33:57 DEBUG { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "instance": "10.142.0.107:9100" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG [...] 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "instance": "10.142.0.43:9100" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG "4761.799999999999" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 ERROR [http_cmd]: ERROR 23:33:57 ERROR --- status code --- 23:33:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:33:57 ERROR --- http response --- 23:33:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:33:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "result": [ 23:33:57 DEBUG { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "instance": "10.142.0.107:9100" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG [...] 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "instance": "10.142.0.43:9100" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG "12192.633333333331" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "result": [ 23:33:57 DEBUG { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG [...] 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG "0" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220437" 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "result": [ 23:33:57 DEBUG { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "pod": "ds-idrepo-2" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG [...] 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "pod": "ds-idrepo-0" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG "0" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220437" 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220437" 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220437" 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220437" 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "result": [ 23:33:57 DEBUG { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG [...] 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG "1447.6055840254462" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:33:57 DEBUG "result": [ 23:33:57 DEBUG "result": [ 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG { 23:33:57 DEBUG { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "pod": "ds-idrepo-2" 23:33:57 DEBUG "pod": "ds-idrepo-2" 23:33:57 DEBUG "data": { 23:33:57 DEBUG }, 23:33:57 DEBUG }, 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "value": [ 23:33:57 DEBUG "value": [ 23:33:57 DEBUG "result": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG { 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "metric": { 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "pod": "ds-idrepo-2" 23:33:57 DEBUG [...] 23:33:57 DEBUG [...] 23:33:57 DEBUG }, 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "pod": "ds-cts-2" 23:33:57 DEBUG "pod": "ds-idrepo-0" 23:33:57 DEBUG 23:33:57 DEBUG }, 23:33:57 DEBUG }, 23:33:57 DEBUG [...] 23:33:57 DEBUG "value": [ 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG 23:33:57 DEBUG "0" 23:33:57 DEBUG "0" 23:33:57 DEBUG 23:33:57 DEBUG ] 23:33:57 DEBUG ] 23:33:57 DEBUG "pod": "ds-idrepo-0" 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 DEBUG }, 23:33:57 DEBUG ] 23:33:57 DEBUG ] 23:33:57 DEBUG "value": [ 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 DEBUG "0" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220437" 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "result": [ 23:33:57 DEBUG { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "pod": "ds-idrepo-2" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG [...] 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "pod": "ds-idrepo-0" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG "0" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220437" 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220437" 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "result": [ 23:33:57 DEBUG { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "instance": "10.142.0.107:9100" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG [...] 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "instance": "10.142.0.43:9100" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG "0.0005666666666665302" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220437" 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220437" 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220437" 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "result": [ 23:33:57 DEBUG { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "instance": "10.142.0.107:9100" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG [...] 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "instance": "10.142.0.43:9100" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG "13913302016" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220437" 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "result": [ 23:33:57 DEBUG { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG [...] 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG "2287.7666666666664" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "result": [ 23:33:57 DEBUG { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG [...] 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG "2447441920" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "result": [ 23:33:57 DEBUG { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "instance": "10.142.0.107:9100" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG [...] 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "instance": "10.142.0.43:9100" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG "0" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:57 INFO [http_cmd]: http status code OK 23:33:57 DEBUG --- status code --- 23:33:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:57 DEBUG --- http response --- 23:33:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:33:57 DEBUG ----- output ----- 23:33:57 DEBUG { 23:33:57 DEBUG "status": "success", 23:33:57 DEBUG "data": { 23:33:57 DEBUG "resultType": "vector", 23:33:57 DEBUG "result": [ 23:33:57 DEBUG { 23:33:57 DEBUG "metric": { 23:33:57 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG [...] 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG 23:33:57 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:33:57 DEBUG }, 23:33:57 DEBUG "value": [ 23:33:57 DEBUG 1713220437, 23:33:57 DEBUG "2288.0666666666666" 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG ] 23:33:57 DEBUG } 23:33:57 DEBUG } 23:33:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220438" 23:33:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220438" 23:33:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220438" 23:33:58 INFO [http_cmd]: http status code OK 23:33:58 DEBUG --- status code --- 23:33:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:58 DEBUG --- http response --- 23:33:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:33:58 DEBUG ----- output ----- 23:33:58 DEBUG { 23:33:58 DEBUG "status": "success", 23:33:58 DEBUG "data": { 23:33:58 DEBUG "resultType": "vector", 23:33:58 DEBUG "result": [ 23:33:58 DEBUG { 23:33:58 DEBUG "metric": { 23:33:58 DEBUG "pod": "ds-idrepo-2" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG [...] 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG "pod": "ds-idrepo-0" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 1713220438, 23:33:58 DEBUG "0" 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG } 23:33:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220438" 23:33:58 INFO [http_cmd]: http status code OK 23:33:58 DEBUG --- status code --- 23:33:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:58 DEBUG --- http response --- 23:33:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:33:58 DEBUG ----- output ----- 23:33:58 DEBUG { 23:33:58 DEBUG "status": "success", 23:33:58 DEBUG "data": { 23:33:58 DEBUG "resultType": "vector", 23:33:58 DEBUG "result": [ 23:33:58 DEBUG { 23:33:58 DEBUG "metric": { 23:33:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG [...] 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 1713220438, 23:33:58 DEBUG "412321" 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG } 23:33:58 INFO [http_cmd]: http status code OK 23:33:58 DEBUG --- status code --- 23:33:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:58 DEBUG --- http response --- 23:33:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:33:58 DEBUG ----- output ----- 23:33:58 DEBUG { 23:33:58 DEBUG "status": "success", 23:33:58 DEBUG "data": { 23:33:58 DEBUG "resultType": "vector", 23:33:58 DEBUG "result": [ 23:33:58 DEBUG { 23:33:58 DEBUG "metric": { 23:33:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG [...] 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG "pod": "am-7d597cf7d6-8vr8f" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 1713220438, 23:33:58 DEBUG "844.4365906815102" 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG } 23:33:58 INFO [http_cmd]: http status code OK 23:33:58 DEBUG --- status code --- 23:33:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:58 DEBUG --- http response --- 23:33:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:33:58 DEBUG ----- output ----- 23:33:58 DEBUG { 23:33:58 DEBUG "status": "success", 23:33:58 DEBUG "data": { 23:33:58 DEBUG "resultType": "vector", 23:33:58 DEBUG "result": [ 23:33:58 DEBUG { 23:33:58 DEBUG "metric": { 23:33:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG [...] 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 1713220438, 23:33:58 DEBUG "0" 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG } 23:33:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220438" 23:33:58 INFO [http_cmd]: http status code OK 23:33:58 DEBUG --- status code --- 23:33:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:58 DEBUG --- http response --- 23:33:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:33:58 DEBUG ----- output ----- 23:33:58 DEBUG { 23:33:58 DEBUG "status": "success", 23:33:58 DEBUG "data": { 23:33:58 DEBUG "resultType": "vector", 23:33:58 DEBUG "result": [ 23:33:58 DEBUG { 23:33:58 DEBUG "metric": { 23:33:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG [...] 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 1713220438, 23:33:58 DEBUG "0.082" 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG } 23:33:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220438" 23:33:58 INFO [http_cmd]: http status code OK 23:33:58 DEBUG --- status code --- 23:33:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:58 DEBUG --- http response --- 23:33:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:33:58 DEBUG ----- output ----- 23:33:58 DEBUG { 23:33:58 DEBUG "status": "success", 23:33:58 DEBUG "data": { 23:33:58 DEBUG "resultType": "vector", 23:33:58 DEBUG "result": [ 23:33:58 DEBUG { 23:33:58 DEBUG "metric": { 23:33:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG [...] 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 1713220438, 23:33:58 DEBUG "6.719280889357587" 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG } 23:33:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220438" 23:33:58 INFO [http_cmd]: http status code OK 23:33:58 DEBUG --- status code --- 23:33:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:58 DEBUG --- http response --- 23:33:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:33:58 DEBUG ----- output ----- 23:33:58 DEBUG { 23:33:58 DEBUG "status": "success", 23:33:58 DEBUG "data": { 23:33:58 DEBUG "resultType": "vector", 23:33:58 DEBUG "result": [ 23:33:58 DEBUG { 23:33:58 DEBUG "metric": { 23:33:58 DEBUG "pod": "ds-idrepo-2" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG [...] 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG "pod": "ds-idrepo-0" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 1713220438, 23:33:58 DEBUG "0" 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG } 23:33:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220438" 23:33:58 INFO [http_cmd]: http status code OK 23:33:58 DEBUG --- status code --- 23:33:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:58 DEBUG --- http response --- 23:33:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:33:58 DEBUG ----- output ----- 23:33:58 DEBUG { 23:33:58 DEBUG "status": "success", 23:33:58 DEBUG "data": { 23:33:58 DEBUG "resultType": "vector", 23:33:58 DEBUG "result": [ 23:33:58 DEBUG { 23:33:58 DEBUG "metric": { 23:33:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG [...] 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 1713220438, 23:33:58 DEBUG "2289.0333333333333" 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG } 23:33:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220438" 23:33:58 INFO [http_cmd]: http status code OK 23:33:58 DEBUG --- status code --- 23:33:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:33:58 DEBUG --- http response --- 23:33:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:33:58 DEBUG ----- output ----- 23:33:58 DEBUG { 23:33:58 DEBUG "status": "success", 23:33:58 DEBUG "data": { 23:33:58 DEBUG "resultType": "vector", 23:33:58 DEBUG "result": [ 23:33:58 DEBUG { 23:33:58 DEBUG "metric": { 23:33:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG [...] 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG 23:33:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:33:58 DEBUG }, 23:33:58 DEBUG "value": [ 23:33:58 DEBUG 1713220438, 23:33:58 DEBUG "2299.8333333333335" 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG ] 23:33:58 DEBUG } 23:33:58 DEBUG } 23:33:59 INFO 23:33:59 INFO [loop_until]: kubectl --namespace=xlou top pods 23:33:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:33:59 INFO [loop_until]: OK (rc = 0) 23:33:59 DEBUG --- stdout --- 23:33:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12827m 8642Mi am-7d597cf7d6-8vr8f 10136m 8911Mi am-7d597cf7d6-9wz9h 15879m 9017Mi ds-cts-0 15163m 8573Mi ds-cts-1 10819m 8657Mi ds-cts-2 10715m 8800Mi ds-idrepo-0 13208m 4988Mi ds-idrepo-1 1816m 2754Mi ds-idrepo-2 1794m 2763Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1026Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6775m 2241Mi 23:33:59 DEBUG --- stderr --- 23:33:59 DEBUG 23:33:59 INFO 23:33:59 INFO [loop_until]: kubectl --namespace=xlou top node 23:33:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:33:59 INFO [loop_until]: OK (rc = 0) 23:33:59 DEBUG --- stdout --- 23:33:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5067Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10521m 35% 10947Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13079m 43% 10717Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17558m 58% 3069Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1948m 6% 5643Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16307m 54% 12926Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2955Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10196m 34% 10416Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1937m 6% 4999Mi 4% gke-xlou-cdm-ds-0f601751-j488 10885m 36% 10474Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2457Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17334m 58% 10669Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13754m 46% 7164Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6857m 43% 6094Mi 10% 23:33:59 DEBUG --- stderr --- 23:33:59 DEBUG 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220497" 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220497" 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220497" 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220497" 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220497" 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220497" 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220497" 23:34:57 INFO [http_cmd]: http status code OK 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220497" 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220497" 23:34:57 DEBUG --- status code --- 23:34:57 ERROR [http_cmd]: ERROR 23:34:57 INFO [http_cmd]: http status code OK 23:34:57 INFO [http_cmd]: http status code OK 23:34:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:57 INFO [http_cmd]: http status code OK 23:34:57 ERROR --- status code --- 23:34:57 DEBUG --- status code --- 23:34:57 DEBUG --- status code --- 23:34:57 DEBUG --- http response --- 23:34:57 DEBUG --- status code --- 23:34:57 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:34:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:34:57 ERROR --- http response --- 23:34:57 DEBUG --- http response --- 23:34:57 DEBUG --- http response --- 23:34:57 DEBUG --- http response --- 23:34:57 DEBUG ----- output ----- 23:34:57 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:34:57 INFO [http_cmd]: http status code OK 23:34:57 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:34:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:34:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:34:57 DEBUG { 23:34:57 INFO [http_cmd]: http status code OK 23:34:57 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:34:57 DEBUG --- status code --- 23:34:57 DEBUG ----- output ----- 23:34:57 DEBUG ----- output ----- 23:34:57 DEBUG "status": "success", 23:34:57 DEBUG --- status code --- 23:34:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:57 DEBUG { 23:34:57 DEBUG { 23:34:57 DEBUG "data": { 23:34:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:57 DEBUG --- http response --- 23:34:57 DEBUG "status": "success", 23:34:57 DEBUG "status": "success", 23:34:57 DEBUG "resultType": "vector", 23:34:57 DEBUG --- http response --- 23:34:57 DEBUG "data": { 23:34:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:34:57 DEBUG "data": { 23:34:57 DEBUG "result": [ 23:34:57 DEBUG "resultType": "vector", 23:34:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:34:57 DEBUG ----- output ----- 23:34:57 DEBUG "resultType": "vector", 23:34:57 DEBUG { 23:34:57 DEBUG "result": [ 23:34:57 DEBUG ----- output ----- 23:34:57 DEBUG { 23:34:57 DEBUG "result": [ 23:34:57 DEBUG "metric": { 23:34:57 DEBUG { 23:34:57 DEBUG { 23:34:57 DEBUG "status": "success", 23:34:57 DEBUG { 23:34:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:34:57 DEBUG "metric": { 23:34:57 DEBUG "status": "success", 23:34:57 DEBUG "data": { 23:34:57 DEBUG "metric": { 23:34:57 DEBUG }, 23:34:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:34:57 DEBUG "data": { 23:34:57 DEBUG "resultType": "vector", 23:34:57 DEBUG "instance": "10.142.0.107:9100" 23:34:57 DEBUG "value": [ 23:34:57 DEBUG }, 23:34:57 DEBUG "resultType": "vector", 23:34:57 DEBUG "result": [ 23:34:57 DEBUG }, 23:34:57 DEBUG 23:34:57 DEBUG "value": [ 23:34:57 DEBUG "result": [ 23:34:57 DEBUG { 23:34:57 DEBUG "value": [ 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG { 23:34:57 DEBUG "metric": { 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG [...] 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220497" 23:34:57 INFO [http_cmd]: http status code OK 23:34:57 DEBUG "metric": { 23:34:57 DEBUG "pod": "ds-idrepo-2" 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG --- status code --- 23:34:57 DEBUG "instance": "10.142.0.107:9100" 23:34:57 INFO [http_cmd]: http status code OK 23:34:57 DEBUG }, 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:57 DEBUG }, 23:34:57 DEBUG --- status code --- 23:34:57 DEBUG "value": [ 23:34:57 DEBUG [...] 23:34:57 DEBUG [...] 23:34:57 DEBUG 23:34:57 DEBUG --- http response --- 23:34:57 DEBUG "value": [ 23:34:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:34:57 DEBUG 23:34:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:34:57 DEBUG --- http response --- 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG }, 23:34:57 DEBUG 23:34:57 DEBUG ----- output ----- 23:34:57 DEBUG 23:34:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG "value": [ 23:34:57 DEBUG 23:34:57 DEBUG { 23:34:57 DEBUG [...] 23:34:57 DEBUG ----- output ----- 23:34:57 DEBUG "instance": "10.142.0.43:9100" 23:34:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:34:57 DEBUG 1713220497, 23:34:57 DEBUG [...] 23:34:57 DEBUG "status": "success", 23:34:57 DEBUG 23:34:57 DEBUG { 23:34:57 DEBUG }, 23:34:57 DEBUG }, 23:34:57 DEBUG "4.41960739672861" 23:34:57 DEBUG 23:34:57 DEBUG "data": { 23:34:57 DEBUG 23:34:57 DEBUG "status": "success", 23:34:57 DEBUG "value": [ 23:34:57 DEBUG "value": [ 23:34:57 DEBUG ] 23:34:57 DEBUG 23:34:57 DEBUG "resultType": "vector", 23:34:57 DEBUG 23:34:57 DEBUG "data": { 23:34:57 DEBUG 1713220497, 23:34:57 DEBUG 1713220497, 23:34:57 DEBUG } 23:34:57 DEBUG 23:34:57 DEBUG "result": [ 23:34:57 DEBUG "pod": "ds-idrepo-0" 23:34:57 DEBUG "resultType": "vector", 23:34:57 DEBUG "3169.7666666666664" 23:34:57 DEBUG "0" 23:34:57 DEBUG ] 23:34:57 DEBUG "instance": "10.142.0.43:9100" 23:34:57 DEBUG { 23:34:57 DEBUG }, 23:34:57 DEBUG "result": [ 23:34:57 DEBUG ] 23:34:57 DEBUG ] 23:34:57 DEBUG } 23:34:57 DEBUG }, 23:34:57 DEBUG "metric": { 23:34:57 DEBUG "value": [ 23:34:57 DEBUG { 23:34:57 DEBUG } 23:34:57 DEBUG } 23:34:57 DEBUG } 23:34:57 DEBUG "value": [ 23:34:57 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:34:57 DEBUG 1713220497, 23:34:57 DEBUG "metric": { 23:34:57 DEBUG ] 23:34:57 DEBUG ] 23:34:57 DEBUG 1713220497, 23:34:57 DEBUG }, 23:34:57 DEBUG "0" 23:34:57 DEBUG "pod": "ds-idrepo-2" 23:34:57 DEBUG } 23:34:57 DEBUG } 23:34:57 DEBUG "11942.866666666665" 23:34:57 DEBUG "value": [ 23:34:57 DEBUG ] 23:34:57 DEBUG }, 23:34:57 DEBUG } 23:34:57 DEBUG } 23:34:57 DEBUG ] 23:34:57 DEBUG 23:34:57 DEBUG } 23:34:57 DEBUG "value": [ 23:34:57 DEBUG } 23:34:57 DEBUG 23:34:57 DEBUG ] 23:34:57 DEBUG 23:34:57 DEBUG ] 23:34:57 DEBUG 23:34:57 DEBUG } 23:34:57 DEBUG 23:34:57 DEBUG } 23:34:57 DEBUG [...] 23:34:57 DEBUG } 23:34:57 DEBUG 23:34:57 DEBUG } 23:34:57 DEBUG 23:34:57 DEBUG [...] 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 INFO [http_cmd]: http status code OK 23:34:57 DEBUG 23:34:57 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:34:57 DEBUG --- status code --- 23:34:57 DEBUG 23:34:57 DEBUG }, 23:34:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:57 DEBUG "pod": "ds-idrepo-0" 23:34:57 DEBUG "value": [ 23:34:57 DEBUG --- http response --- 23:34:57 DEBUG }, 23:34:57 DEBUG 1713220497, 23:34:57 DEBUG "value": [ 23:34:57 DEBUG "0" 23:34:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:34:57 DEBUG 1713220497, 23:34:57 DEBUG ] 23:34:57 DEBUG ----- output ----- 23:34:57 DEBUG "0" 23:34:57 DEBUG } 23:34:57 DEBUG { 23:34:57 DEBUG ] 23:34:57 DEBUG ] 23:34:57 DEBUG "status": "success", 23:34:57 DEBUG } 23:34:57 DEBUG } 23:34:57 DEBUG "data": { 23:34:57 DEBUG ] 23:34:57 DEBUG } 23:34:57 DEBUG "resultType": "vector", 23:34:57 DEBUG } 23:34:57 DEBUG "result": [ 23:34:57 DEBUG } 23:34:57 DEBUG { 23:34:57 DEBUG "metric": { 23:34:57 DEBUG "pod": "ds-idrepo-2" 23:34:57 DEBUG }, 23:34:57 DEBUG "value": [ 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG [...] 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG "pod": "ds-idrepo-0" 23:34:57 DEBUG }, 23:34:57 DEBUG "value": [ 23:34:57 DEBUG 1713220497, 23:34:57 DEBUG "0" 23:34:57 DEBUG ] 23:34:57 DEBUG } 23:34:57 DEBUG ] 23:34:57 DEBUG } 23:34:57 DEBUG } 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220497" 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220497" 23:34:57 INFO [http_cmd]: http status code OK 23:34:57 DEBUG --- status code --- 23:34:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:57 DEBUG --- http response --- 23:34:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:34:57 DEBUG ----- output ----- 23:34:57 DEBUG { 23:34:57 DEBUG "status": "success", 23:34:57 DEBUG "data": { 23:34:57 DEBUG "resultType": "vector", 23:34:57 DEBUG "result": [ 23:34:57 DEBUG { 23:34:57 DEBUG "metric": { 23:34:57 DEBUG "pod": "ds-idrepo-2" 23:34:57 DEBUG }, 23:34:57 DEBUG "value": [ 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG [...] 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG "pod": "ds-cts-2" 23:34:57 DEBUG }, 23:34:57 DEBUG "value": [ 23:34:57 DEBUG 1713220497, 23:34:57 DEBUG "0" 23:34:57 DEBUG ] 23:34:57 DEBUG } 23:34:57 DEBUG ] 23:34:57 DEBUG } 23:34:57 DEBUG } 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220497" 23:34:57 INFO [http_cmd]: http status code OK 23:34:57 DEBUG --- status code --- 23:34:57 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:57 DEBUG --- http response --- 23:34:57 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:34:57 DEBUG ----- output ----- 23:34:57 DEBUG { 23:34:57 DEBUG "status": "success", 23:34:57 DEBUG "data": { 23:34:57 DEBUG "resultType": "vector", 23:34:57 DEBUG "result": [ 23:34:57 DEBUG { 23:34:57 DEBUG "metric": { 23:34:57 DEBUG "instance": "10.142.0.107:9100" 23:34:57 DEBUG }, 23:34:57 DEBUG "value": [ 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG [...] 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG 23:34:57 DEBUG "instance": "10.142.0.43:9100" 23:34:57 DEBUG }, 23:34:57 DEBUG "value": [ 23:34:57 DEBUG 1713220497, 23:34:57 DEBUG "0" 23:34:57 DEBUG ] 23:34:57 DEBUG } 23:34:57 DEBUG ] 23:34:57 DEBUG } 23:34:57 DEBUG } 23:34:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220497" 23:34:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220498" 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG { 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG "data": { 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "result": [ 23:34:58 DEBUG { 23:34:58 DEBUG "metric": { 23:34:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG [...] 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220497, 23:34:58 DEBUG "2448809984" 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG } 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG { 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG "data": { 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "result": [ 23:34:58 DEBUG { 23:34:58 DEBUG "metric": { 23:34:58 DEBUG "pod": "ds-idrepo-2" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG [...] 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG "pod": "ds-idrepo-0" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220498, 23:34:58 DEBUG "0" 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG } 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG { 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG "data": { 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "result": [ 23:34:58 DEBUG { 23:34:58 DEBUG "metric": { 23:34:58 DEBUG "instance": "10.142.0.107:9100" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG [...] 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG "instance": "10.142.0.43:9100" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220497, 23:34:58 DEBUG "0.00036666666666652263" 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG } 23:34:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220498" 23:34:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220498" 23:34:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220498" 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG { 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG "data": { 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "result": [ 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220498" 23:34:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220498" 23:34:58 DEBUG { 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG "metric": { 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG "instance": "10.142.0.107:9100" 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG }, 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:34:58 DEBUG "value": [ 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG 23:34:58 DEBUG { 23:34:58 DEBUG 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG 23:34:58 DEBUG "data": { 23:34:58 DEBUG [...] 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG 23:34:58 DEBUG "result": [ 23:34:58 DEBUG 23:34:58 DEBUG { 23:34:58 DEBUG 23:34:58 DEBUG "metric": { 23:34:58 DEBUG "instance": "10.142.0.43:9100" 23:34:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:34:58 DEBUG }, 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220498, 23:34:58 DEBUG 23:34:58 DEBUG "13914340352" 23:34:58 DEBUG 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG ] 23:34:58 DEBUG 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG } 23:34:58 DEBUG [...] 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG ] 23:34:58 DEBUG 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG } 23:34:58 DEBUG 23:34:58 DEBUG } 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:34:58 DEBUG 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:34:58 DEBUG { 23:34:58 DEBUG }, 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG "value": [ 23:34:58 DEBUG "data": { 23:34:58 DEBUG 1713220498, 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "0" 23:34:58 DEBUG "result": [ 23:34:58 DEBUG ] 23:34:58 DEBUG { 23:34:58 DEBUG } 23:34:58 DEBUG "metric": { 23:34:58 DEBUG ] 23:34:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:34:58 DEBUG } 23:34:58 DEBUG }, 23:34:58 DEBUG } 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG [...] 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220498, 23:34:58 DEBUG "2299.9333333333334" 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG } 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG { 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG "data": { 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "result": [ 23:34:58 DEBUG { 23:34:58 DEBUG "metric": { 23:34:58 DEBUG "pod": "ds-idrepo-2" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG [...] 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG "pod": "ds-idrepo-0" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220498, 23:34:58 DEBUG "0" 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG } 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG { 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG "data": { 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "result": [ 23:34:58 DEBUG { 23:34:58 DEBUG "metric": { 23:34:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG [...] 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220498, 23:34:58 DEBUG "2297.9333333333334" 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG } 23:34:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220498" 23:34:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220498" 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG { 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG "data": { 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "result": [ 23:34:58 DEBUG { 23:34:58 DEBUG "metric": { 23:34:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG [...] 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220498, 23:34:58 DEBUG "412285" 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG } 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG { 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG "data": { 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "result": [ 23:34:58 DEBUG { 23:34:58 DEBUG "metric": { 23:34:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG [...] 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220498, 23:34:58 DEBUG "0" 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG } 23:34:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220498" 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG { 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG "data": { 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "result": [ 23:34:58 DEBUG { 23:34:58 DEBUG "metric": { 23:34:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG [...] 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220498, 23:34:58 DEBUG "0.082" 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG } 23:34:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220498" 23:34:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220498" 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG { 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG "data": { 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "result": [ 23:34:58 DEBUG { 23:34:58 DEBUG "metric": { 23:34:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG [...] 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220498, 23:34:58 DEBUG "6.713385233800135" 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG } 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG { 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG "data": { 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "result": [ 23:34:58 DEBUG { 23:34:58 DEBUG "metric": { 23:34:58 DEBUG "pod": "ds-idrepo-2" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG [...] 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG "pod": "ds-idrepo-0" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220498, 23:34:58 DEBUG "0" 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG } 23:34:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220498" 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG { 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG "data": { 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "result": [ 23:34:58 DEBUG { 23:34:58 DEBUG "metric": { 23:34:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG [...] 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220498, 23:34:58 DEBUG "2295.9333333333334" 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG } 23:34:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220498" 23:34:58 INFO [http_cmd]: http status code OK 23:34:58 DEBUG --- status code --- 23:34:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:34:58 DEBUG --- http response --- 23:34:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:34:58 DEBUG ----- output ----- 23:34:58 DEBUG { 23:34:58 DEBUG "status": "success", 23:34:58 DEBUG "data": { 23:34:58 DEBUG "resultType": "vector", 23:34:58 DEBUG "result": [ 23:34:58 DEBUG { 23:34:58 DEBUG "metric": { 23:34:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG [...] 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG 23:34:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:34:58 DEBUG }, 23:34:58 DEBUG "value": [ 23:34:58 DEBUG 1713220498, 23:34:58 DEBUG "2289.233333333333" 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG ] 23:34:58 DEBUG } 23:34:58 DEBUG } 23:34:59 INFO 23:34:59 INFO [loop_until]: kubectl --namespace=xlou top pods 23:34:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:34:59 INFO [loop_until]: OK (rc = 0) 23:34:59 DEBUG --- stdout --- 23:34:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12393m 8639Mi am-7d597cf7d6-8vr8f 10091m 8910Mi am-7d597cf7d6-9wz9h 15973m 8997Mi ds-cts-0 15661m 8574Mi ds-cts-1 10953m 8669Mi ds-cts-2 10955m 8809Mi ds-idrepo-0 13398m 4988Mi ds-idrepo-1 1817m 2754Mi ds-idrepo-2 1790m 2763Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6598m 2243Mi 23:34:59 DEBUG --- stderr --- 23:34:59 DEBUG 23:34:59 INFO 23:34:59 INFO [loop_until]: kubectl --namespace=xlou top node 23:34:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:34:59 INFO [loop_until]: OK (rc = 0) 23:34:59 DEBUG --- stdout --- 23:34:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5082Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10950m 36% 10955Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13059m 43% 10715Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17592m 58% 3060Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1952m 6% 5631Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16774m 56% 12916Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2968Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10381m 34% 10415Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1940m 6% 5009Mi 4% gke-xlou-cdm-ds-0f601751-j488 10856m 36% 10481Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2464Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17428m 58% 10652Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13783m 46% 7167Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6924m 43% 6092Mi 10% 23:34:59 DEBUG --- stderr --- 23:34:59 DEBUG 23:35:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220557" 23:35:57 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220557" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG ----- output ----- 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG { 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG "data": { 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG "result": [ 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG --- http response --- 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG { 23:35:58 DEBUG { 23:35:58 DEBUG { 23:35:58 DEBUG { 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG "data": { 23:35:58 DEBUG "pod": "ds-idrepo-2" 23:35:58 DEBUG "data": { 23:35:58 DEBUG "data": { 23:35:58 ERROR [http_cmd]: ERROR 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG { 23:35:58 DEBUG { 23:35:58 DEBUG { 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG }, 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220558" 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "resultType": "vector", 23:35:58 ERROR --- status code --- 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:35:58 DEBUG "result": [ 23:35:58 DEBUG "value": [ 23:35:58 DEBUG "result": [ 23:35:58 DEBUG "result": [ 23:35:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:35:58 DEBUG "data": { 23:35:58 DEBUG "data": { 23:35:58 DEBUG "data": { 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG 23:35:58 DEBUG { 23:35:58 DEBUG { 23:35:58 ERROR --- http response --- 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:35:58 DEBUG { 23:35:58 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:35:58 DEBUG "result": [ 23:35:58 DEBUG "result": [ 23:35:58 DEBUG "result": [ 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "pod": "ds-idrepo-2" 23:35:58 DEBUG 23:35:58 DEBUG "pod": "ds-idrepo-2" 23:35:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:35:58 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG { 23:35:58 DEBUG "data": { 23:35:58 DEBUG }, 23:35:58 DEBUG [...] 23:35:58 DEBUG }, 23:35:58 DEBUG }, 23:35:58 DEBUG "data": { 23:35:58 DEBUG { 23:35:58 DEBUG "instance": "10.142.0.107:9100" 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG "value": [ 23:35:58 DEBUG "value": [ 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "metric": { 23:35:58 DEBUG }, 23:35:58 DEBUG "pod": "ds-idrepo-2" 23:35:58 DEBUG "result": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "result": [ 23:35:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:35:58 DEBUG "value": [ 23:35:58 DEBUG }, 23:35:58 DEBUG { 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG { 23:35:58 DEBUG }, 23:35:58 DEBUG 23:35:58 DEBUG "value": [ 23:35:58 DEBUG "metric": { 23:35:58 DEBUG 23:35:58 DEBUG "pod": "ds-idrepo-0" 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "instance": "10.142.0.107:9100" 23:35:58 DEBUG [...] 23:35:58 DEBUG }, 23:35:58 DEBUG [...] 23:35:58 DEBUG [...] 23:35:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG }, 23:35:58 DEBUG 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG }, 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 1713220557, 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "0" 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "pod": "ds-idrepo-0" 23:35:58 DEBUG ] 23:35:58 DEBUG "pod": "ds-cts-2" 23:35:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG }, 23:35:58 DEBUG } 23:35:58 DEBUG }, 23:35:58 DEBUG }, 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "instance": "10.142.0.43:9100" 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG "value": [ 23:35:58 DEBUG ] 23:35:58 DEBUG "value": [ 23:35:58 DEBUG "value": [ 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG }, 23:35:58 DEBUG "pod": "ds-idrepo-0" 23:35:58 DEBUG 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG } 23:35:58 DEBUG 1713220557, 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG 23:35:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:35:58 DEBUG "value": [ 23:35:58 DEBUG }, 23:35:58 DEBUG 23:35:58 DEBUG "0" 23:35:58 DEBUG } 23:35:58 DEBUG "0" 23:35:58 DEBUG "0" 23:35:58 DEBUG 23:35:58 DEBUG }, 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG ] 23:35:58 DEBUG ] 23:35:58 DEBUG ] 23:35:58 DEBUG 23:35:58 DEBUG "value": [ 23:35:58 DEBUG "0.00033333333333333365" 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "instance": "10.142.0.43:9100" 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG ] 23:35:58 DEBUG "0" 23:35:58 DEBUG ] 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG ] 23:35:58 DEBUG ] 23:35:58 DEBUG }, 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 DEBUG "0" 23:35:58 DEBUG "value": [ 23:35:58 DEBUG ] 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG "2884.5333333333333" 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "6.672233198635635" 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG { 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG } 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "instance": "10.142.0.107:9100" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "instance": "10.142.0.43:9100" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "12005.566666666664" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "2296.4333333333334" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220558" 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "pod": "ds-idrepo-2" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "pod": "ds-idrepo-0" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "0" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "2450403328" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "2296.1666666666665" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "instance": "10.142.0.107:9100" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "instance": "10.142.0.43:9100" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "0.000500000000000019" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "instance": "10.142.0.107:9100" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "instance": "10.142.0.43:9100" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "13914979328" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "pod": "ds-idrepo-2" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "pod": "ds-idrepo-0" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "0" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "71935843.46337289" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "412430" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "0" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "0.082" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220558" 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "6.748098066518376" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "pod": "ds-idrepo-2" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "pod": "ds-idrepo-0" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "0" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "2295.9666666666667" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220558" 23:35:58 INFO [http_cmd]: http status code OK 23:35:58 DEBUG --- status code --- 23:35:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:35:58 DEBUG --- http response --- 23:35:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:35:58 DEBUG ----- output ----- 23:35:58 DEBUG { 23:35:58 DEBUG "status": "success", 23:35:58 DEBUG "data": { 23:35:58 DEBUG "resultType": "vector", 23:35:58 DEBUG "result": [ 23:35:58 DEBUG { 23:35:58 DEBUG "metric": { 23:35:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG [...] 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG 23:35:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:35:58 DEBUG }, 23:35:58 DEBUG "value": [ 23:35:58 DEBUG 1713220558, 23:35:58 DEBUG "2287.7666666666664" 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG ] 23:35:58 DEBUG } 23:35:58 DEBUG } 23:35:59 INFO 23:35:59 INFO [loop_until]: kubectl --namespace=xlou top pods 23:35:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:35:59 INFO [loop_until]: OK (rc = 0) 23:35:59 DEBUG --- stdout --- 23:35:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12940m 8638Mi am-7d597cf7d6-8vr8f 10182m 8911Mi am-7d597cf7d6-9wz9h 16058m 8995Mi ds-cts-0 15624m 8569Mi ds-cts-1 10822m 8648Mi ds-cts-2 10851m 8804Mi ds-idrepo-0 13521m 4989Mi ds-idrepo-1 1827m 2754Mi ds-idrepo-2 1830m 2763Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1026Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6679m 2243Mi 23:35:59 DEBUG --- stderr --- 23:35:59 DEBUG 23:35:59 INFO 23:35:59 INFO [loop_until]: kubectl --namespace=xlou top node 23:35:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:35:59 INFO [loop_until]: OK (rc = 0) 23:35:59 DEBUG --- stdout --- 23:35:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5067Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10895m 36% 10966Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13269m 44% 10717Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17555m 58% 3060Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1938m 6% 5642Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16618m 55% 12923Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2975Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10261m 34% 10415Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1936m 6% 5009Mi 4% gke-xlou-cdm-ds-0f601751-j488 10681m 35% 10483Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2449Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17356m 58% 10645Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13764m 46% 7167Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6842m 43% 6093Mi 10% 23:35:59 DEBUG --- stderr --- 23:35:59 DEBUG 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220618" 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "pod": "ds-idrepo-2" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "ds-idrepo-0" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "0" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG --- http response --- 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 ERROR [http_cmd]: ERROR 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG --- status code --- 23:36:58 ERROR --- status code --- 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG { 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG { 23:36:58 DEBUG { 23:36:58 DEBUG { 23:36:58 DEBUG { 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG "status": "success", 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- http response --- 23:36:58 ERROR --- http response --- 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:36:58 DEBUG "data": { 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:36:58 DEBUG "data": { 23:36:58 DEBUG "data": { 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG { 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "result": [ 23:36:58 DEBUG "result": [ 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG { 23:36:58 DEBUG { 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "data": { 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "data": { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG { 23:36:58 DEBUG "instance": "10.142.0.107:9100" 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "pod": "ds-idrepo-2" 23:36:58 DEBUG "instance": "10.142.0.107:9100" 23:36:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:36:58 DEBUG }, 23:36:58 DEBUG "result": [ 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG }, 23:36:58 DEBUG "result": [ 23:36:58 DEBUG }, 23:36:58 DEBUG }, 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG { 23:36:58 DEBUG "data": { 23:36:58 DEBUG "value": [ 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG { 23:36:58 DEBUG "value": [ 23:36:58 DEBUG "value": [ 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG "metric": { 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "ds-idrepo-2" 23:36:58 DEBUG "result": [ 23:36:58 DEBUG 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG }, 23:36:58 DEBUG { 23:36:58 DEBUG 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG }, 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG "value": [ 23:36:58 DEBUG "metric": { 23:36:58 DEBUG [...] 23:36:58 DEBUG "value": [ 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:36:58 DEBUG [...] 23:36:58 DEBUG [...] 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "ds-idrepo-2" 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG }, 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG { 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG "instance": "10.142.0.43:9100" 23:36:58 DEBUG [...] 23:36:58 DEBUG "data": { 23:36:58 DEBUG "pod": "ds-idrepo-0" 23:36:58 DEBUG "instance": "10.142.0.43:9100" 23:36:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:36:58 DEBUG }, 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG }, 23:36:58 DEBUG 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG }, 23:36:58 DEBUG }, 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG "result": [ 23:36:58 DEBUG "value": [ 23:36:58 DEBUG "value": [ 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG 23:36:58 DEBUG { 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "0.005726985363368749" 23:36:58 DEBUG "pod": "ds-idrepo-0" 23:36:58 DEBUG 23:36:58 DEBUG "11995.93333333333" 23:36:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "0" 23:36:58 DEBUG "3104.5" 23:36:58 DEBUG "0" 23:36:58 DEBUG ] 23:36:58 DEBUG }, 23:36:58 DEBUG 23:36:58 DEBUG ] 23:36:58 DEBUG }, 23:36:58 DEBUG "instance": "10.142.0.107:9100" 23:36:58 DEBUG ] 23:36:58 DEBUG ] 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG } 23:36:58 DEBUG "value": [ 23:36:58 DEBUG }, 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "pod": "ds-cts-2" 23:36:58 DEBUG ] 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG ] 23:36:58 DEBUG ] 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG "0" 23:36:58 DEBUG }, 23:36:58 DEBUG } 23:36:58 DEBUG "597.4494844347241" 23:36:58 DEBUG 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG "value": [ 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG } 23:36:58 DEBUG 23:36:58 DEBUG ] 23:36:58 DEBUG "0" 23:36:58 DEBUG ] 23:36:58 DEBUG [...] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 DEBUG 23:36:58 DEBUG ] 23:36:58 DEBUG 23:36:58 DEBUG } 23:36:58 DEBUG "instance": "10.142.0.43:9100" 23:36:58 DEBUG } 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "0" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "pod": "ds-idrepo-2" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "ds-idrepo-0" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "0" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "pod": "ds-idrepo-2" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "ds-idrepo-0" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "0" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220618" 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "2287.1666666666665" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220618" 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:36:58 DEBUG "instance": "10.142.0.107:9100" 23:36:58 DEBUG }, 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:36:58 DEBUG "instance": "10.142.0.43:9100" 23:36:58 DEBUG }, 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "2287.2" 23:36:58 DEBUG "13915702272" 23:36:58 DEBUG ] 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "2451918848" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "instance": "10.142.0.107:9100" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "instance": "10.142.0.43:9100" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "0.00040000000000001514" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "45101681.17639973" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "412869.5" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "0" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "0.082" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220618" 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "6.709332998178058" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "pod": "ds-idrepo-2" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "ds-idrepo-0" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "0" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "2288.6666666666665" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220618" 23:36:58 INFO [http_cmd]: http status code OK 23:36:58 DEBUG --- status code --- 23:36:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:36:58 DEBUG --- http response --- 23:36:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:36:58 DEBUG ----- output ----- 23:36:58 DEBUG { 23:36:58 DEBUG "status": "success", 23:36:58 DEBUG "data": { 23:36:58 DEBUG "resultType": "vector", 23:36:58 DEBUG "result": [ 23:36:58 DEBUG { 23:36:58 DEBUG "metric": { 23:36:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG [...] 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG 23:36:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:36:58 DEBUG }, 23:36:58 DEBUG "value": [ 23:36:58 DEBUG 1713220618, 23:36:58 DEBUG "2286.5" 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG ] 23:36:58 DEBUG } 23:36:58 DEBUG } 23:36:59 INFO 23:36:59 INFO [loop_until]: kubectl --namespace=xlou top pods 23:36:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:36:59 INFO [loop_until]: OK (rc = 0) 23:36:59 DEBUG --- stdout --- 23:36:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12839m 8639Mi am-7d597cf7d6-8vr8f 10052m 8912Mi am-7d597cf7d6-9wz9h 15426m 9004Mi ds-cts-0 15131m 8583Mi ds-cts-1 10990m 8652Mi ds-cts-2 10676m 8798Mi ds-idrepo-0 13400m 4988Mi ds-idrepo-1 1848m 2755Mi ds-idrepo-2 1820m 2762Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6587m 2246Mi 23:36:59 DEBUG --- stderr --- 23:36:59 DEBUG 23:36:59 INFO 23:36:59 INFO [loop_until]: kubectl --namespace=xlou top node 23:36:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:36:59 INFO [loop_until]: OK (rc = 0) 23:36:59 DEBUG --- stdout --- 23:36:59 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5060Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10985m 36% 10958Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13196m 44% 10720Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17508m 58% 3057Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1947m 6% 5647Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16463m 55% 12912Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2961Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10299m 34% 10422Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1978m 6% 5018Mi 4% gke-xlou-cdm-ds-0f601751-j488 10694m 35% 10495Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2453Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17308m 57% 10658Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13736m 46% 7169Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6860m 43% 6094Mi 10% 23:36:59 DEBUG --- stderr --- 23:36:59 DEBUG 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220678" 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "ds-idrepo-2" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "1.8234788709832368" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220678" 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "instance": "10.142.0.107:9100" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "instance": "10.142.0.43:9100" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "0" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "ds-idrepo-2" 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "ds-idrepo-2" 23:37:58 DEBUG }, 23:37:58 DEBUG "instance": "10.142.0.107:9100" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "ds-idrepo-0" 23:37:58 DEBUG 23:37:58 DEBUG "instance": "10.142.0.43:9100" 23:37:58 DEBUG }, 23:37:58 DEBUG "pod": "ds-cts-2" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG }, 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 DEBUG "value": [ 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 ERROR [http_cmd]: ERROR 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG "0" 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "3105.5" 23:37:58 ERROR --- status code --- 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG ] 23:37:58 DEBUG "0" 23:37:58 DEBUG ] 23:37:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 ERROR --- http response --- 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:37:58 DEBUG } 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:37:58 DEBUG } 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 DEBUG { 23:37:58 DEBUG } 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "instance": "10.142.0.107:9100" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "instance": "10.142.0.43:9100" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "12311.566666666664" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "ds-idrepo-2" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "ds-idrepo-0" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "0" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "ds-idrepo-2" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "524.9094928395219" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "ds-idrepo-2" 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "ds-idrepo-2" 23:37:58 DEBUG }, 23:37:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "ds-idrepo-0" 23:37:58 DEBUG 23:37:58 DEBUG "pod": "ds-idrepo-0" 23:37:58 DEBUG }, 23:37:58 DEBUG "pod": "ds-idrepo-2" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "0" 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "0" 23:37:58 DEBUG ] 23:37:58 DEBUG "5380.3223016051" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "instance": "10.142.0.107:9100" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "instance": "10.142.0.43:9100" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "13916455936" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "ds-idrepo-2" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "ds-idrepo-0" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "0" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "instance": "10.142.0.107:9100" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "instance": "10.142.0.43:9100" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "0.0004666666666665264" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "2302.1666666666665" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220678" 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "2303.6" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "2453123072" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG [...] 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG 23:37:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "413196.5" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "ds-idrepo-2" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "6430.141287284144" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "0" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "0.082" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220678" 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "6.660985897547949" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "ds-idrepo-2" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "ds-idrepo-0" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "0" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "2296.5" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220678" 23:37:58 INFO [http_cmd]: http status code OK 23:37:58 DEBUG --- status code --- 23:37:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:37:58 DEBUG --- http response --- 23:37:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:37:58 DEBUG ----- output ----- 23:37:58 DEBUG { 23:37:58 DEBUG "status": "success", 23:37:58 DEBUG "data": { 23:37:58 DEBUG "resultType": "vector", 23:37:58 DEBUG "result": [ 23:37:58 DEBUG { 23:37:58 DEBUG "metric": { 23:37:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG [...] 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG 23:37:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:37:58 DEBUG }, 23:37:58 DEBUG "value": [ 23:37:58 DEBUG 1713220678, 23:37:58 DEBUG "2303.0666666666666" 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG ] 23:37:58 DEBUG } 23:37:58 DEBUG } 23:37:59 INFO 23:37:59 INFO [loop_until]: kubectl --namespace=xlou top pods 23:37:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:37:59 INFO [loop_until]: OK (rc = 0) 23:37:59 DEBUG --- stdout --- 23:37:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12781m 8638Mi am-7d597cf7d6-8vr8f 10044m 8910Mi am-7d597cf7d6-9wz9h 16229m 8996Mi ds-cts-0 15262m 8571Mi ds-cts-1 11087m 8654Mi ds-cts-2 10645m 8805Mi ds-idrepo-0 13516m 4988Mi ds-idrepo-1 1788m 2755Mi ds-idrepo-2 1830m 2762Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6619m 2247Mi 23:37:59 DEBUG --- stderr --- 23:37:59 DEBUG 23:38:00 INFO 23:38:00 INFO [loop_until]: kubectl --namespace=xlou top node 23:38:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:38:00 INFO [loop_until]: OK (rc = 0) 23:38:00 DEBUG --- stdout --- 23:38:00 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 59m 0% 5079Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11010m 36% 10972Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13232m 44% 10720Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17612m 58% 3067Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1943m 6% 5646Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16495m 55% 12918Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2968Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10381m 34% 10421Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1939m 6% 5016Mi 4% gke-xlou-cdm-ds-0f601751-j488 10727m 35% 10490Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2449Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17386m 58% 10652Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13809m 46% 7163Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6842m 43% 6090Mi 10% 23:38:00 DEBUG --- stderr --- 23:38:00 DEBUG 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220738" 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG { 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG "data": { 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG ----- output ----- 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 DEBUG "result": [ 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG { 23:38:58 DEBUG { 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG { 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG --- status code --- 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "pod": "ds-idrepo-2" 23:38:58 DEBUG "data": { 23:38:58 DEBUG "data": { 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG "instance": "10.142.0.107:9100" 23:38:58 DEBUG "instance": "10.142.0.107:9100" 23:38:58 DEBUG }, 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG }, 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG "result": [ 23:38:58 DEBUG "result": [ 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:38:58 DEBUG "value": [ 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG { 23:38:58 DEBUG { 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:38:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG }, 23:38:58 DEBUG }, 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG [...] 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 ERROR [http_cmd]: ERROR 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 ERROR --- status code --- 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG "pod": "ds-idrepo-0" 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "instance": "10.142.0.43:9100" 23:38:58 DEBUG "instance": "10.142.0.43:9100" 23:38:58 ERROR --- http response --- 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG }, 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:38:58 DEBUG }, 23:38:58 DEBUG }, 23:38:58 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:38:58 DEBUG "value": [ 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG "pod": "ds-idrepo-2" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG "value": [ 23:38:58 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG { 23:38:58 DEBUG { 23:38:58 DEBUG "0" 23:38:58 DEBUG 23:38:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG 23:38:58 DEBUG "12426.233333333332" 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG ] 23:38:58 DEBUG 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG }, 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG 23:38:58 DEBUG "4577.233333333333" 23:38:58 DEBUG 23:38:58 DEBUG ] 23:38:58 DEBUG "data": { 23:38:58 DEBUG "data": { 23:38:58 DEBUG } 23:38:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG "value": [ 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG 23:38:58 DEBUG ] 23:38:58 DEBUG [...] 23:38:58 DEBUG } 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG ] 23:38:58 DEBUG }, 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG 23:38:58 DEBUG } 23:38:58 DEBUG 23:38:58 DEBUG ] 23:38:58 DEBUG "result": [ 23:38:58 DEBUG "result": [ 23:38:58 DEBUG } 23:38:58 DEBUG "value": [ 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG "0" 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG { 23:38:58 DEBUG { 23:38:58 DEBUG } 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG ] 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "pod": "ds-idrepo-2" 23:38:58 DEBUG } 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG } 23:38:58 DEBUG "instance": "10.142.0.107:9100" 23:38:58 DEBUG "365.3554544643653" 23:38:58 DEBUG }, 23:38:58 DEBUG ] 23:38:58 DEBUG { 23:38:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:38:58 DEBUG "pod": "ds-idrepo-0" 23:38:58 DEBUG { 23:38:58 DEBUG }, 23:38:58 DEBUG ] 23:38:58 DEBUG "value": [ 23:38:58 DEBUG } 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG }, 23:38:58 DEBUG }, 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "value": [ 23:38:58 DEBUG } 23:38:58 DEBUG 23:38:58 DEBUG } 23:38:58 DEBUG "data": { 23:38:58 DEBUG "value": [ 23:38:58 DEBUG "value": [ 23:38:58 DEBUG "data": { 23:38:58 DEBUG 23:38:58 DEBUG ] 23:38:58 DEBUG 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "0.003118514554157151" 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG 23:38:58 DEBUG } 23:38:58 DEBUG 23:38:58 DEBUG "result": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG ] 23:38:58 DEBUG "result": [ 23:38:58 DEBUG 23:38:58 DEBUG } 23:38:58 DEBUG [...] 23:38:58 DEBUG { 23:38:58 DEBUG "0" 23:38:58 DEBUG } 23:38:58 DEBUG { 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG "metric": { 23:38:58 DEBUG ] 23:38:58 DEBUG ] 23:38:58 DEBUG "metric": { 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "pod": "ds-idrepo-2" 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 DEBUG "pod": "ds-idrepo-2" 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG }, 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG }, 23:38:58 DEBUG 23:38:58 DEBUG "pod": "ds-idrepo-0" 23:38:58 DEBUG "value": [ 23:38:58 DEBUG } 23:38:58 DEBUG "value": [ 23:38:58 DEBUG "instance": "10.142.0.43:9100" 23:38:58 DEBUG }, 23:38:58 DEBUG 23:38:58 DEBUG } 23:38:58 DEBUG 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "0" 23:38:58 DEBUG [...] 23:38:58 DEBUG [...] 23:38:58 DEBUG "0.0003333333333333299" 23:38:58 DEBUG ] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG "pod": "ds-idrepo-0" 23:38:58 DEBUG "pod": "ds-cts-2" 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 DEBUG }, 23:38:58 DEBUG }, 23:38:58 DEBUG } 23:38:58 DEBUG "value": [ 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "0" 23:38:58 DEBUG "0" 23:38:58 DEBUG ] 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "2297.8" 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "instance": "10.142.0.107:9100" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "instance": "10.142.0.43:9100" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "13917295616" 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "pod": "ds-idrepo-2" 23:38:58 DEBUG "instance": "10.142.0.107:9100" 23:38:58 DEBUG }, 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:38:58 DEBUG "pod": "ds-idrepo-0" 23:38:58 DEBUG "instance": "10.142.0.43:9100" 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG }, 23:38:58 DEBUG }, 23:38:58 DEBUG { 23:38:58 DEBUG "value": [ 23:38:58 DEBUG "value": [ 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "data": { 23:38:58 DEBUG "0" 23:38:58 DEBUG "0.00033333333333350387" 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG ] 23:38:58 DEBUG ] 23:38:58 DEBUG "result": [ 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 DEBUG { 23:38:58 DEBUG ] 23:38:58 DEBUG ] 23:38:58 DEBUG "metric": { 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "2297.3" 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220738" 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "2455433216" 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "412918" 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "32147870.00862248" 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "0" 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "0.082" 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220738" 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "pod": "ds-idrepo-2" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "pod": "ds-idrepo-0" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "0" 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "6.6655445922992165" 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "2298.4" 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220738" 23:38:58 INFO [http_cmd]: http status code OK 23:38:58 DEBUG --- status code --- 23:38:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:38:58 DEBUG --- http response --- 23:38:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:38:58 DEBUG ----- output ----- 23:38:58 DEBUG { 23:38:58 DEBUG "status": "success", 23:38:58 DEBUG "data": { 23:38:58 DEBUG "resultType": "vector", 23:38:58 DEBUG "result": [ 23:38:58 DEBUG { 23:38:58 DEBUG "metric": { 23:38:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG [...] 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG 23:38:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:38:58 DEBUG }, 23:38:58 DEBUG "value": [ 23:38:58 DEBUG 1713220738, 23:38:58 DEBUG "2296.3" 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG ] 23:38:58 DEBUG } 23:38:58 DEBUG } 23:38:59 INFO 23:38:59 INFO [loop_until]: kubectl --namespace=xlou top pods 23:38:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:38:59 INFO [loop_until]: OK (rc = 0) 23:38:59 DEBUG --- stdout --- 23:38:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12473m 8640Mi am-7d597cf7d6-8vr8f 9954m 8911Mi am-7d597cf7d6-9wz9h 15763m 8994Mi ds-cts-0 15737m 8582Mi ds-cts-1 11006m 8682Mi ds-cts-2 10443m 8802Mi ds-idrepo-0 13263m 4989Mi ds-idrepo-1 1840m 2755Mi ds-idrepo-2 1847m 2762Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6688m 2249Mi 23:38:59 DEBUG --- stderr --- 23:38:59 DEBUG 23:39:00 INFO 23:39:00 INFO [loop_until]: kubectl --namespace=xlou top node 23:39:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:39:00 INFO [loop_until]: OK (rc = 0) 23:39:00 DEBUG --- stdout --- 23:39:00 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5072Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10852m 36% 10974Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13049m 43% 10716Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17572m 58% 3056Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1948m 6% 5643Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16559m 55% 12944Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2966Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 9912m 33% 10417Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1925m 6% 5015Mi 4% gke-xlou-cdm-ds-0f601751-j488 10581m 35% 10495Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2453Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17448m 58% 10639Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13864m 46% 7172Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6610m 41% 6094Mi 10% 23:39:00 DEBUG --- stderr --- 23:39:00 DEBUG 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220798" 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG { 23:39:58 DEBUG { 23:39:58 DEBUG { 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "data": { 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:39:58 DEBUG "data": { 23:39:58 DEBUG "data": { 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "result": [ 23:39:58 DEBUG "result": [ 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG { 23:39:58 DEBUG { 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "data": { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "pod": "ds-idrepo-2" 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "pod": "ds-idrepo-2" 23:39:58 DEBUG "pod": "ds-idrepo-2" 23:39:58 DEBUG "instance": "10.142.0.107:9100" 23:39:58 DEBUG }, 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG }, 23:39:58 DEBUG "result": [ 23:39:58 DEBUG }, 23:39:58 DEBUG }, 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG "value": [ 23:39:58 DEBUG { 23:39:58 DEBUG "value": [ 23:39:58 DEBUG "value": [ 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:39:58 DEBUG 23:39:58 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:39:58 DEBUG "metric": { 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG 23:39:58 DEBUG "instance": "10.142.0.107:9100" 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG { 23:39:58 DEBUG 23:39:58 DEBUG }, 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG [...] 23:39:58 DEBUG "value": [ 23:39:58 DEBUG [...] 23:39:58 DEBUG [...] 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220798" 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "result": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG { 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "pod": "ds-idrepo-0" 23:39:58 DEBUG 23:39:58 DEBUG "metric": { 23:39:58 DEBUG [...] 23:39:58 DEBUG "pod": "ds-idrepo-0" 23:39:58 DEBUG "pod": "ds-idrepo-0" 23:39:58 DEBUG "instance": "10.142.0.43:9100" 23:39:58 DEBUG }, 23:39:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:39:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:39:58 DEBUG 23:39:58 DEBUG }, 23:39:58 DEBUG }, 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG }, 23:39:58 DEBUG }, 23:39:58 DEBUG 23:39:58 DEBUG "value": [ 23:39:58 DEBUG "value": [ 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "0" 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG 23:39:58 DEBUG "instance": "10.142.0.43:9100" 23:39:58 DEBUG "0" 23:39:58 DEBUG "0" 23:39:58 DEBUG "0" 23:39:58 DEBUG ] 23:39:58 DEBUG "0" 23:39:58 DEBUG 23:39:58 DEBUG }, 23:39:58 DEBUG ] 23:39:58 DEBUG ] 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG 23:39:58 DEBUG "value": [ 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG [...] 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG ] 23:39:58 DEBUG ] 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG 23:39:58 DEBUG "12340.199999999997" 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG 23:39:58 DEBUG } 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:39:58 DEBUG ] 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:39:58 DEBUG }, 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG "value": [ 23:39:58 DEBUG } 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG { 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "0" 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG "data": { 23:39:58 DEBUG ] 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:39:58 DEBUG } 23:39:58 DEBUG "result": [ 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG ] 23:39:58 DEBUG { 23:39:58 DEBUG { 23:39:58 DEBUG } 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG } 23:39:58 DEBUG "instance": "10.142.0.107:9100" 23:39:58 DEBUG "data": { 23:39:58 DEBUG }, 23:39:58 DEBUG "resultType": "vector", 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 DEBUG "value": [ 23:39:58 DEBUG "result": [ 23:39:58 DEBUG 23:39:58 DEBUG { 23:39:58 DEBUG 23:39:58 DEBUG "metric": { 23:39:58 DEBUG 23:39:58 DEBUG "pod": "ds-idrepo-2" 23:39:58 DEBUG [...] 23:39:58 DEBUG }, 23:39:58 DEBUG 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "instance": "10.142.0.43:9100" 23:39:58 DEBUG 23:39:58 DEBUG }, 23:39:58 DEBUG [...] 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG 23:39:58 DEBUG "6422.2" 23:39:58 DEBUG 23:39:58 DEBUG ] 23:39:58 DEBUG "pod": "ds-cts-2" 23:39:58 DEBUG } 23:39:58 DEBUG }, 23:39:58 DEBUG ] 23:39:58 DEBUG "value": [ 23:39:58 DEBUG } 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG } 23:39:58 DEBUG "0" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 ERROR [http_cmd]: ERROR 23:39:58 ERROR --- status code --- 23:39:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:39:58 ERROR --- http response --- 23:39:58 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:39:58 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "3.4417650407294627" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "pod": "ds-idrepo-2" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "pod": "ds-idrepo-0" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "0" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "2305.866666666667" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "instance": "10.142.0.107:9100" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "instance": "10.142.0.43:9100" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "13917815808" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG { 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "data": { 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG "result": [ 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG { 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:39:58 DEBUG "pod": "ds-idrepo-2" 23:39:58 DEBUG "instance": "10.142.0.107:9100" 23:39:58 DEBUG }, 23:39:58 DEBUG }, 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG "value": [ 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG [...] 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:39:58 DEBUG "pod": "ds-idrepo-0" 23:39:58 DEBUG "instance": "10.142.0.43:9100" 23:39:58 DEBUG }, 23:39:58 DEBUG }, 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG "value": [ 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "2305.133333333333" 23:39:58 DEBUG "0" 23:39:58 DEBUG "0.00036666666666652263" 23:39:58 DEBUG ] 23:39:58 DEBUG ] 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG ] 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220798" 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "2456379392" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "64280630.729357116" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "412986" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "0" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "0.082" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220798" 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "pod": "ds-idrepo-2" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "pod": "ds-idrepo-0" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "0" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "6.691724152721833" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "2305.233333333333" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220798" 23:39:58 INFO [http_cmd]: http status code OK 23:39:58 DEBUG --- status code --- 23:39:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:39:58 DEBUG --- http response --- 23:39:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:39:58 DEBUG ----- output ----- 23:39:58 DEBUG { 23:39:58 DEBUG "status": "success", 23:39:58 DEBUG "data": { 23:39:58 DEBUG "resultType": "vector", 23:39:58 DEBUG "result": [ 23:39:58 DEBUG { 23:39:58 DEBUG "metric": { 23:39:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG [...] 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG 23:39:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:39:58 DEBUG }, 23:39:58 DEBUG "value": [ 23:39:58 DEBUG 1713220798, 23:39:58 DEBUG "2289.866666666667" 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG ] 23:39:58 DEBUG } 23:39:58 DEBUG } 23:39:59 INFO 23:39:59 INFO [loop_until]: kubectl --namespace=xlou top pods 23:39:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:39:59 INFO [loop_until]: OK (rc = 0) 23:39:59 DEBUG --- stdout --- 23:39:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12222m 8675Mi am-7d597cf7d6-8vr8f 10148m 8911Mi am-7d597cf7d6-9wz9h 15981m 8997Mi ds-cts-0 15820m 8588Mi ds-cts-1 10393m 8674Mi ds-cts-2 10935m 8805Mi ds-idrepo-0 13538m 4988Mi ds-idrepo-1 1688m 2755Mi ds-idrepo-2 1833m 2763Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 5657m 1562Mi 23:39:59 DEBUG --- stderr --- 23:39:59 DEBUG 23:40:00 INFO 23:40:00 INFO [loop_until]: kubectl --namespace=xlou top node 23:40:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:40:00 INFO [loop_until]: OK (rc = 0) 23:40:00 DEBUG --- stdout --- 23:40:00 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5074Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10689m 35% 10968Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13185m 44% 10746Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 14426m 48% 3054Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1945m 6% 5639Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16430m 55% 12926Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2965Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10276m 34% 10415Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1851m 6% 5011Mi 4% gke-xlou-cdm-ds-0f601751-j488 10751m 36% 10502Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2454Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 15470m 51% 10655Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13081m 43% 7168Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6868m 43% 6092Mi 10% 23:40:00 DEBUG --- stderr --- 23:40:00 DEBUG 23:40:15 WARNING File /tmp/lodemon_data-20240415_230746/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [15/Apr/2024 23:40:15] "GET /monitoring/average?start_time=20240415_221100&stop_time=20240415_223814 HTTP/1.1" 200 - 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220858" 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG --- http response --- 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220858" 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG ----- output ----- 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220858" 23:40:58 DEBUG --- status code --- 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG { 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220858" 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG --- http response --- 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG "data": { 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG "result": [ 23:40:58 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG { 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG { 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG "status": "success", 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:40:58 DEBUG "metric": { 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:40:58 DEBUG "data": { 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 ERROR [http_cmd]: ERROR 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG ----- output ----- 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG ----- output ----- 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG "pod": "ds-idrepo-2" 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG --- status code --- 23:40:58 ERROR --- status code --- 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG { 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG { 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG }, 23:40:58 DEBUG { 23:40:58 DEBUG "result": [ 23:40:58 DEBUG { 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG "value": [ 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG { 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG --- http response --- 23:40:58 ERROR --- http response --- 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG "data": { 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG "data": { 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG 23:40:58 DEBUG "data": { 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "data": { 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:40:58 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:40:58 DEBUG 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG ----- output ----- 23:40:58 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:40:58 DEBUG "result": [ 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG "result": [ 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG "result": [ 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG }, 23:40:58 DEBUG "result": [ 23:40:58 DEBUG { 23:40:58 DEBUG { 23:40:58 DEBUG { 23:40:58 DEBUG { 23:40:58 DEBUG { 23:40:58 DEBUG { 23:40:58 DEBUG [...] 23:40:58 DEBUG { 23:40:58 DEBUG { 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG "value": [ 23:40:58 DEBUG { 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG "metric": { 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "data": { 23:40:58 DEBUG "pod": "ds-idrepo-2" 23:40:58 DEBUG "data": { 23:40:58 DEBUG "data": { 23:40:58 DEBUG "pod": "ds-idrepo-2" 23:40:58 DEBUG "data": { 23:40:58 DEBUG 23:40:58 DEBUG "data": { 23:40:58 DEBUG "pod": "ds-idrepo-2" 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG 23:40:58 DEBUG "instance": "10.142.0.107:9100" 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG }, 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG }, 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "result": [ 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG }, 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:40:58 DEBUG 23:40:58 DEBUG }, 23:40:58 DEBUG "result": [ 23:40:58 DEBUG "value": [ 23:40:58 DEBUG "result": [ 23:40:58 DEBUG "result": [ 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG "pod": "ds-idrepo-0" 23:40:58 DEBUG "result": [ 23:40:58 DEBUG "value": [ 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG [...] 23:40:58 DEBUG "value": [ 23:40:58 DEBUG { 23:40:58 DEBUG 23:40:58 DEBUG { 23:40:58 DEBUG { 23:40:58 DEBUG 23:40:58 DEBUG { 23:40:58 DEBUG }, 23:40:58 DEBUG { 23:40:58 DEBUG 23:40:58 DEBUG { 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "metric": { 23:40:58 DEBUG 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "metric": { 23:40:58 DEBUG 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "value": [ 23:40:58 DEBUG "metric": { 23:40:58 DEBUG 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "instance": "10.142.0.107:9100" 23:40:58 DEBUG 23:40:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:40:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:40:58 DEBUG 23:40:58 DEBUG "instance": "10.142.0.107:9100" 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG "pod": "ds-idrepo-2" 23:40:58 DEBUG 23:40:58 DEBUG "data": { 23:40:58 DEBUG { 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG }, 23:40:58 DEBUG [...] 23:40:58 DEBUG }, 23:40:58 DEBUG }, 23:40:58 DEBUG [...] 23:40:58 DEBUG }, 23:40:58 DEBUG "0" 23:40:58 DEBUG }, 23:40:58 DEBUG [...] 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:40:58 DEBUG [...] 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG "value": [ 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG "value": [ 23:40:58 DEBUG ] 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG "result": [ 23:40:58 DEBUG "data": { 23:40:58 DEBUG }, 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG } 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG { 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG ] 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "result": [ 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "pod": "ds-idrepo-0" 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "pod": "ds-idrepo-0" 23:40:58 DEBUG 23:40:58 DEBUG } 23:40:58 DEBUG 23:40:58 DEBUG [...] 23:40:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:40:58 DEBUG { 23:40:58 DEBUG "0.005601129560839644" 23:40:58 DEBUG "instance": "10.142.0.43:9100" 23:40:58 DEBUG [...] 23:40:58 DEBUG }, 23:40:58 DEBUG [...] 23:40:58 DEBUG [...] 23:40:58 DEBUG }, 23:40:58 DEBUG [...] 23:40:58 DEBUG } 23:40:58 DEBUG "pod": "ds-cts-2" 23:40:58 DEBUG 23:40:58 DEBUG }, 23:40:58 DEBUG "metric": { 23:40:58 DEBUG ] 23:40:58 DEBUG }, 23:40:58 DEBUG 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG }, 23:40:58 DEBUG 23:40:58 DEBUG "value": [ 23:40:58 DEBUG "pod": "ds-idrepo-2" 23:40:58 DEBUG } 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG }, 23:40:58 DEBUG ] 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG 23:40:58 DEBUG "0" 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "0" 23:40:58 DEBUG 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG "pod": "ds-idrepo-0" 23:40:58 DEBUG 23:40:58 DEBUG "value": [ 23:40:58 DEBUG } 23:40:58 DEBUG "0" 23:40:58 DEBUG "instance": "10.142.0.43:9100" 23:40:58 DEBUG ] 23:40:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:40:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:40:58 DEBUG ] 23:40:58 DEBUG "instance": "10.142.0.43:9100" 23:40:58 DEBUG "0" 23:40:58 DEBUG }, 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG }, 23:40:58 DEBUG } 23:40:58 DEBUG }, 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG }, 23:40:58 DEBUG ] 23:40:58 DEBUG "value": [ 23:40:58 DEBUG [...] 23:40:58 DEBUG 23:40:58 DEBUG } 23:40:58 DEBUG "value": [ 23:40:58 DEBUG ] 23:40:58 DEBUG "value": [ 23:40:58 DEBUG } 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG } 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG ] 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG } 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG ] 23:40:58 DEBUG "1207.2429375462452" 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG ] 23:40:58 DEBUG "0" 23:40:58 DEBUG 23:40:58 DEBUG [...] 23:40:58 DEBUG } 23:40:58 DEBUG "5527.366666666667" 23:40:58 DEBUG } 23:40:58 DEBUG "0" 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG "12458.099999999999" 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:40:58 DEBUG 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG }, 23:40:58 DEBUG 23:40:58 DEBUG ] 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG "value": [ 23:40:58 DEBUG "pod": "ds-idrepo-0" 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG }, 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG "242.1" 23:40:58 DEBUG "value": [ 23:40:58 DEBUG ] 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG } 23:40:58 DEBUG "0" 23:40:58 DEBUG ] 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220858" 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220858" 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG { 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG "data": { 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "result": [ 23:40:58 DEBUG { 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "instance": "10.142.0.107:9100" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG [...] 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "instance": "10.142.0.43:9100" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG "13918491648" 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG --- http response --- 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG { 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:40:58 DEBUG "data": { 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG { 23:40:58 DEBUG "result": [ 23:40:58 DEBUG { 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG { 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG "data": { 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "data": { 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "result": [ 23:40:58 DEBUG }, 23:40:58 DEBUG "result": [ 23:40:58 DEBUG { 23:40:58 DEBUG "value": [ 23:40:58 DEBUG { 23:40:58 DEBUG "metric": { 23:40:58 DEBUG 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "instance": "10.142.0.107:9100" 23:40:58 DEBUG 23:40:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:40:58 DEBUG }, 23:40:58 DEBUG 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG [...] 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG [...] 23:40:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:40:58 DEBUG [...] 23:40:58 DEBUG 23:40:58 DEBUG }, 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG 23:40:58 DEBUG "instance": "10.142.0.43:9100" 23:40:58 DEBUG "318944.5" 23:40:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:40:58 DEBUG }, 23:40:58 DEBUG ] 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG } 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG ] 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG "0.00040000000000001514" 23:40:58 DEBUG } 23:40:58 DEBUG "241.46666666666667" 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220858" 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG { 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG "data": { 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "result": [ 23:40:58 DEBUG { 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG [...] 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG "2021109760" 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:40:58 DEBUG ----- output ----- 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG { 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG "data": { 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "result": [ 23:40:58 DEBUG { 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:40:58 DEBUG 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG [...] 23:40:58 DEBUG { 23:40:58 DEBUG 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG 23:40:58 DEBUG "data": { 23:40:58 DEBUG 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:40:58 DEBUG "result": [ 23:40:58 DEBUG }, 23:40:58 DEBUG { 23:40:58 DEBUG "value": [ 23:40:58 DEBUG "metric": { 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:40:58 DEBUG "0" 23:40:58 DEBUG }, 23:40:58 DEBUG ] 23:40:58 DEBUG "value": [ 23:40:58 DEBUG } 23:40:58 DEBUG 23:40:58 DEBUG ] 23:40:58 DEBUG 23:40:58 DEBUG } 23:40:58 DEBUG 23:40:58 DEBUG } 23:40:58 DEBUG [...] 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG "79453707.23736145" 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG { 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG "data": { 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "result": [ 23:40:58 DEBUG { 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG [...] 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG "0.082" 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG { 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG "data": { 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "result": [ 23:40:58 DEBUG { 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "pod": "ds-idrepo-2" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG [...] 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "pod": "ds-idrepo-0" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG "0" 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220858" 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG { 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG "data": { 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "result": [ 23:40:58 DEBUG { 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG [...] 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG "0.3720350729873893" 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG { 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG "data": { 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "result": [ 23:40:58 DEBUG { 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG [...] 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG "242.16666666666666" 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220858" 23:40:58 INFO [http_cmd]: http status code OK 23:40:58 DEBUG --- status code --- 23:40:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:40:58 DEBUG --- http response --- 23:40:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:40:58 DEBUG ----- output ----- 23:40:58 DEBUG { 23:40:58 DEBUG "status": "success", 23:40:58 DEBUG "data": { 23:40:58 DEBUG "resultType": "vector", 23:40:58 DEBUG "result": [ 23:40:58 DEBUG { 23:40:58 DEBUG "metric": { 23:40:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG [...] 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG 23:40:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:40:58 DEBUG }, 23:40:58 DEBUG "value": [ 23:40:58 DEBUG 1713220858, 23:40:58 DEBUG "2300.1666666666665" 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG ] 23:40:58 DEBUG } 23:40:58 DEBUG } 23:40:59 INFO 23:40:59 INFO [loop_until]: kubectl --namespace=xlou top pods 23:40:59 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:40:59 INFO [loop_until]: OK (rc = 0) 23:40:59 DEBUG --- stdout --- 23:40:59 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 3334m 8642Mi am-7d597cf7d6-8vr8f 2033m 8910Mi am-7d597cf7d6-9wz9h 4189m 9033Mi ds-cts-0 7771m 8544Mi ds-cts-1 6286m 8683Mi ds-cts-2 5946m 8801Mi ds-idrepo-0 1414m 4989Mi ds-idrepo-1 523m 2755Mi ds-idrepo-2 456m 2763Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 3357m 1993Mi 23:40:59 DEBUG --- stderr --- 23:40:59 DEBUG 23:41:00 INFO 23:41:00 INFO [loop_until]: kubectl --namespace=xlou top node 23:41:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:41:00 INFO [loop_until]: OK (rc = 0) 23:41:00 DEBUG --- stdout --- 23:41:00 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5073Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5991m 20% 11001Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 3449m 11% 10742Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 4098m 13% 3042Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 388m 1% 5654Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 8989m 30% 12920Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2961Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 2497m 8% 10428Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 524m 1% 5013Mi 4% gke-xlou-cdm-ds-0f601751-j488 5945m 19% 10496Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2450Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 5903m 19% 10683Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 3509m 11% 7186Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 3295m 20% 5836Mi 9% 23:41:00 DEBUG --- stderr --- 23:41:00 DEBUG 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:41:58 DEBUG ----- output ----- 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG { 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "data": { 23:41:58 DEBUG --- http response --- 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220918" 23:41:58 DEBUG "data": { 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG "resultType": "vector", 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 ERROR [http_cmd]: ERROR 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG "result": [ 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG --- status code --- 23:41:58 ERROR --- status code --- 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG { 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG { 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG "metric": { 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG --- http response --- 23:41:58 ERROR --- http response --- 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "data": { 23:41:58 DEBUG "pod": "ds-idrepo-2" 23:41:58 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG "data": { 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:41:58 DEBUG "pod": "ds-idrepo-2" 23:41:58 DEBUG }, 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG }, 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:41:58 DEBUG "value": [ 23:41:58 DEBUG "result": [ 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220918" 23:41:58 DEBUG "value": [ 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG { 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG { 23:41:58 DEBUG { 23:41:58 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:41:58 DEBUG 23:41:58 DEBUG { 23:41:58 DEBUG 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG 23:41:58 DEBUG "metric": { 23:41:58 DEBUG 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "data": { 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:41:58 DEBUG "data": { 23:41:58 DEBUG { 23:41:58 DEBUG "data": { 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG "data": { 23:41:58 DEBUG 23:41:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:41:58 DEBUG 23:41:58 DEBUG "data": { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG [...] 23:41:58 DEBUG }, 23:41:58 DEBUG [...] 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG "result": [ 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "pod": "ds-idrepo-2" 23:41:58 DEBUG "metric": { 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG { 23:41:58 DEBUG "data": { 23:41:58 DEBUG { 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "pod": "ds-idrepo-0" 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "instance": "10.142.0.107:9100" 23:41:58 DEBUG }, 23:41:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "metric": { 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG }, 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "data": { 23:41:58 DEBUG "value": [ 23:41:58 DEBUG }, 23:41:58 DEBUG }, 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:41:58 DEBUG "result": [ 23:41:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "value": [ 23:41:58 DEBUG "instance": "10.142.0.107:9100" 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG 23:41:58 DEBUG "value": [ 23:41:58 DEBUG "value": [ 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG }, 23:41:58 DEBUG { 23:41:58 DEBUG "metric": { 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG "pod": "ds-cts-2" 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG }, 23:41:58 DEBUG "result": [ 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "value": [ 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:41:58 DEBUG }, 23:41:58 DEBUG "pod": "ds-idrepo-2" 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG [...] 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG }, 23:41:58 DEBUG "0" 23:41:58 DEBUG "value": [ 23:41:58 DEBUG { 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG "value": [ 23:41:58 DEBUG }, 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG "value": [ 23:41:58 DEBUG ] 23:41:58 DEBUG 23:41:58 DEBUG "metric": { 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG [...] 23:41:58 DEBUG { 23:41:58 DEBUG 23:41:58 DEBUG "value": [ 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "0" 23:41:58 DEBUG 23:41:58 DEBUG "pod": "ds-idrepo-2" 23:41:58 DEBUG 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:41:58 DEBUG 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG }, 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "pod": "ds-idrepo-2" 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG [...] 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG "value": [ 23:41:58 DEBUG "instance": "10.142.0.43:9100" 23:41:58 DEBUG 23:41:58 DEBUG "data": { 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG }, 23:41:58 DEBUG { 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 DEBUG { 23:41:58 DEBUG 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG 23:41:58 DEBUG }, 23:41:58 DEBUG 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG 23:41:58 DEBUG "pod": "ds-idrepo-2" 23:41:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:41:58 DEBUG "value": [ 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG ] 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG [...] 23:41:58 DEBUG { 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG }, 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "data": { 23:41:58 DEBUG } 23:41:58 DEBUG "data": { 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:41:58 DEBUG "pod": "ds-idrepo-0" 23:41:58 DEBUG 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "instance": "10.142.0.107:9100" 23:41:58 DEBUG 23:41:58 DEBUG "value": [ 23:41:58 DEBUG "5314.766757389709" 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG } 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "instance": "10.142.0.43:9100" 23:41:58 DEBUG [...] 23:41:58 DEBUG "13919235072" 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG }, 23:41:58 DEBUG 23:41:58 DEBUG }, 23:41:58 DEBUG }, 23:41:58 DEBUG 23:41:58 DEBUG "pod": "ds-idrepo-0" 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "result": [ 23:41:58 DEBUG "result": [ 23:41:58 DEBUG }, 23:41:58 DEBUG 23:41:58 DEBUG ] 23:41:58 DEBUG { 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG "value": [ 23:41:58 DEBUG "value": [ 23:41:58 DEBUG ] 23:41:58 DEBUG }, 23:41:58 DEBUG "2483.5666666666666" 23:41:58 DEBUG { 23:41:58 DEBUG { 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG } 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "pod": "ds-idrepo-2" 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG 23:41:58 DEBUG } 23:41:58 DEBUG "value": [ 23:41:58 DEBUG ] 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "metric": { 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG 23:41:58 DEBUG ] 23:41:58 DEBUG "data": { 23:41:58 DEBUG "0" 23:41:58 DEBUG }, 23:41:58 DEBUG "0" 23:41:58 DEBUG 23:41:58 DEBUG ] 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG } 23:41:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:41:58 DEBUG "instance": "10.142.0.107:9100" 23:41:58 DEBUG "5690.966666666666" 23:41:58 DEBUG "pod": "ds-idrepo-0" 23:41:58 DEBUG } 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG ] 23:41:58 DEBUG "value": [ 23:41:58 DEBUG ] 23:41:58 DEBUG 23:41:58 DEBUG } 23:41:58 DEBUG "0" 23:41:58 DEBUG ] 23:41:58 DEBUG }, 23:41:58 DEBUG }, 23:41:58 DEBUG ] 23:41:58 DEBUG }, 23:41:58 DEBUG } 23:41:58 DEBUG "result": [ 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG [...] 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG "value": [ 23:41:58 DEBUG "value": [ 23:41:58 DEBUG } 23:41:58 DEBUG "value": [ 23:41:58 DEBUG { 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 DEBUG "1.9077471991898065" 23:41:58 DEBUG 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG ] 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "metric": { 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG 23:41:58 DEBUG ] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG } 23:41:58 DEBUG "0" 23:41:58 DEBUG "pod": "ds-idrepo-2" 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 DEBUG 23:41:58 DEBUG } 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG }, 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG "instance": "10.142.0.43:9100" 23:41:58 DEBUG } 23:41:58 DEBUG [...] 23:41:58 DEBUG [...] 23:41:58 DEBUG } 23:41:58 DEBUG "value": [ 23:41:58 DEBUG } 23:41:58 DEBUG }, 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG ] 23:41:58 DEBUG 23:41:58 DEBUG } 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG } 23:41:58 DEBUG 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG } 23:41:58 DEBUG 23:41:58 DEBUG "0.000333333333333341" 23:41:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:41:58 DEBUG "instance": "10.142.0.43:9100" 23:41:58 DEBUG [...] 23:41:58 DEBUG ] 23:41:58 DEBUG }, 23:41:58 DEBUG }, 23:41:58 DEBUG 23:41:58 DEBUG } 23:41:58 DEBUG "value": [ 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG ] 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG 23:41:58 DEBUG } 23:41:58 DEBUG "2483.9" 23:41:58 DEBUG "12625.099999999999" 23:41:58 DEBUG "pod": "ds-idrepo-0" 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG }, 23:41:58 DEBUG ] 23:41:58 DEBUG ] 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 DEBUG "value": [ 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG "0" 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "data": { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "instance": "10.142.0.107:9100" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "instance": "10.142.0.43:9100" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "0.000500000000000019" 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "data": { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "282015" 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220918" 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "data": { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "2840403968" 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "data": { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "0" 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "data": { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "pod": "ds-idrepo-2" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "119182866.76371923" 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "data": { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "0.082" 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "data": { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "pod": "ds-idrepo-2" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "pod": "ds-idrepo-0" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "0" 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220918" 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "data": { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "7.433669193689167" 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "data": { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "2484.133333333333" 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG } 23:41:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220918" 23:41:58 INFO [http_cmd]: http status code OK 23:41:58 DEBUG --- status code --- 23:41:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:41:58 DEBUG --- http response --- 23:41:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:41:58 DEBUG ----- output ----- 23:41:58 DEBUG { 23:41:58 DEBUG "status": "success", 23:41:58 DEBUG "data": { 23:41:58 DEBUG "resultType": "vector", 23:41:58 DEBUG "result": [ 23:41:58 DEBUG { 23:41:58 DEBUG "metric": { 23:41:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG [...] 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG 23:41:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:41:58 DEBUG }, 23:41:58 DEBUG "value": [ 23:41:58 DEBUG 1713220918, 23:41:58 DEBUG "2378.7" 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG ] 23:41:58 DEBUG } 23:41:58 DEBUG } 23:42:00 INFO 23:42:00 INFO [loop_until]: kubectl --namespace=xlou top pods 23:42:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:42:00 INFO [loop_until]: OK (rc = 0) 23:42:00 DEBUG --- stdout --- 23:42:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13832m 8643Mi am-7d597cf7d6-8vr8f 10779m 8910Mi am-7d597cf7d6-9wz9h 17515m 9018Mi ds-cts-0 15599m 8581Mi ds-cts-1 11157m 8674Mi ds-cts-2 10610m 8800Mi ds-idrepo-0 14013m 4989Mi ds-idrepo-1 1973m 2755Mi ds-idrepo-2 1925m 2763Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7806m 2984Mi 23:42:00 DEBUG --- stderr --- 23:42:00 DEBUG 23:42:00 INFO 23:42:00 INFO [loop_until]: kubectl --namespace=xlou top node 23:42:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:42:00 INFO [loop_until]: OK (rc = 0) 23:42:00 DEBUG --- stdout --- 23:42:00 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5078Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11084m 37% 10981Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 14213m 47% 10722Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 20488m 68% 3358Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 2071m 6% 5644Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16470m 55% 12929Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2976Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 11102m 37% 10422Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2092m 7% 5001Mi 4% gke-xlou-cdm-ds-0f601751-j488 10705m 35% 10491Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2444Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 18395m 61% 10794Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14655m 49% 7170Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 7984m 50% 6804Mi 11% 23:42:00 DEBUG --- stderr --- 23:42:00 DEBUG 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 ERROR [http_cmd]: ERROR 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 ERROR --- status code --- 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 DEBUG --- status code --- 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG --- status code --- 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220978" 23:42:58 DEBUG --- status code --- 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713220978" 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG --- status code --- 23:42:58 ERROR --- http response --- 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713220978" 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG --- http response --- 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG --- status code --- 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG --- http response --- 23:42:58 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:42:58 DEBUG --- status code --- 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG { 23:42:58 DEBUG { 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG { 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG { 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG { 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG { 23:42:58 DEBUG { 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "data": { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG { 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG { 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG "data": { 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:42:58 DEBUG "data": { 23:42:58 DEBUG "result": [ 23:42:58 DEBUG "data": { 23:42:58 DEBUG { 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG { 23:42:58 DEBUG { 23:42:58 DEBUG { 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG "data": { 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG "data": { 23:42:58 DEBUG "data": { 23:42:58 DEBUG { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "data": { 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "data": { 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "data": { 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "result": [ 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "result": [ 23:42:58 DEBUG { 23:42:58 DEBUG "result": [ 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "result": [ 23:42:58 DEBUG { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "data": { 23:42:58 DEBUG "data": { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "data": { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG { 23:42:58 DEBUG "data": { 23:42:58 DEBUG { 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG { 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "result": [ 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "result": [ 23:42:58 DEBUG { 23:42:58 DEBUG "result": [ 23:42:58 DEBUG "data": { 23:42:58 DEBUG "result": [ 23:42:58 DEBUG { 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "data": { 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG "pod": "ds-idrepo-2" 23:42:58 DEBUG }, 23:42:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:42:58 DEBUG { 23:42:58 DEBUG "result": [ 23:42:58 DEBUG "result": [ 23:42:58 DEBUG { 23:42:58 DEBUG "metric": { 23:42:58 DEBUG { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG { 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "data": { 23:42:58 DEBUG "pod": "ds-idrepo-2" 23:42:58 DEBUG "result": [ 23:42:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "value": [ 23:42:58 DEBUG }, 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:42:58 DEBUG }, 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "pod": "ds-idrepo-2" 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG "pod": "ds-idrepo-2" 23:42:58 DEBUG }, 23:42:58 DEBUG "result": [ 23:42:58 DEBUG "result": [ 23:42:58 DEBUG { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG }, 23:42:58 DEBUG { 23:42:58 DEBUG }, 23:42:58 DEBUG "result": [ 23:42:58 DEBUG 23:42:58 DEBUG "value": [ 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG "value": [ 23:42:58 DEBUG { 23:42:58 DEBUG "result": [ 23:42:58 DEBUG "metric": { 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "value": [ 23:42:58 DEBUG { 23:42:58 DEBUG "instance": "10.142.0.107:9100" 23:42:58 DEBUG }, 23:42:58 DEBUG "result": [ 23:42:58 DEBUG "value": [ 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "value": [ 23:42:58 DEBUG { 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG { 23:42:58 DEBUG 23:42:58 DEBUG "metric": { 23:42:58 DEBUG { 23:42:58 DEBUG "pod": "ds-idrepo-2" 23:42:58 DEBUG }, 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "value": [ 23:42:58 DEBUG "metric": { 23:42:58 DEBUG { 23:42:58 DEBUG 23:42:58 DEBUG "instance": "10.142.0.107:9100" 23:42:58 DEBUG 23:42:58 DEBUG "metric": { 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG 23:42:58 DEBUG "instance": "10.142.0.107:9100" 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "value": [ 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG "pod": "ds-idrepo-2" 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "metric": { 23:42:58 DEBUG 23:42:58 DEBUG }, 23:42:58 DEBUG 23:42:58 DEBUG "instance": "10.142.0.107:9100" 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG "data": { 23:42:58 DEBUG 23:42:58 DEBUG }, 23:42:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:42:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG }, 23:42:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:42:58 DEBUG 23:42:58 DEBUG "instance": "10.142.0.107:9100" 23:42:58 DEBUG 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG }, 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG [...] 23:42:58 DEBUG "value": [ 23:42:58 DEBUG }, 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "value": [ 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "result": [ 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "pod": "ds-idrepo-0" 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "instance": "10.142.0.43:9100" 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG { 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG }, 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG }, 23:42:58 DEBUG 23:42:58 DEBUG "pod": "ds-cts-2" 23:42:58 DEBUG 23:42:58 DEBUG "metric": { 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG 23:42:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:42:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:42:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG }, 23:42:58 DEBUG "6045.233333333333" 23:42:58 DEBUG [...] 23:42:58 DEBUG }, 23:42:58 DEBUG }, 23:42:58 DEBUG }, 23:42:58 DEBUG [...] 23:42:58 DEBUG [...] 23:42:58 DEBUG "value": [ 23:42:58 DEBUG "instance": "10.142.0.43:9100" 23:42:58 DEBUG "pod": "ds-idrepo-0" 23:42:58 DEBUG }, 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "pod": "ds-idrepo-2" 23:42:58 DEBUG }, 23:42:58 DEBUG 23:42:58 DEBUG "value": [ 23:42:58 DEBUG "value": [ 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG }, 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG "value": [ 23:42:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "0" 23:42:58 DEBUG "value": [ 23:42:58 DEBUG "value": [ 23:42:58 DEBUG "pod": "ds-idrepo-2" 23:42:58 DEBUG "pod": "ds-idrepo-0" 23:42:58 DEBUG }, 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "value": [ 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG }, 23:42:58 DEBUG "275434.5" 23:42:58 DEBUG ] 23:42:58 DEBUG "1096.9666666666667" 23:42:58 DEBUG 23:42:58 DEBUG "pod": "ds-idrepo-0" 23:42:58 DEBUG ] 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG }, 23:42:58 DEBUG "pod": "ds-idrepo-0" 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG "instance": "10.142.0.43:9100" 23:42:58 DEBUG 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG 23:42:58 DEBUG "pod": "ds-idrepo-2" 23:42:58 DEBUG }, 23:42:58 DEBUG "13920185344" 23:42:58 DEBUG "0" 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG "1.6554162601103972" 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG "0" 23:42:58 DEBUG "0" 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG ] 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG ] 23:42:58 DEBUG }, 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG "instance": "10.142.0.43:9100" 23:42:58 DEBUG ] 23:42:58 DEBUG ] 23:42:58 DEBUG "1096.9666666666667" 23:42:58 DEBUG [...] 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG ] 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG } 23:42:58 DEBUG "instance": "10.142.0.43:9100" 23:42:58 DEBUG }, 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG "0" 23:42:58 DEBUG }, 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG "5258.698164077545" 23:42:58 DEBUG ] 23:42:58 DEBUG "0" 23:42:58 DEBUG "value": [ 23:42:58 DEBUG "value": [ 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG "value": [ 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG "0" 23:42:58 DEBUG ] 23:42:58 DEBUG ] 23:42:58 DEBUG ] 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG "0.00043333333333303397" 23:42:58 DEBUG "0.00033333333333333365" 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG "13015.999999999998" 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG ] 23:42:58 DEBUG ] 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG }, 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG "value": [ 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG "3321806848" 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG { 23:42:58 DEBUG { 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "data": { 23:42:58 DEBUG "data": { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "result": [ 23:42:58 DEBUG "result": [ 23:42:58 DEBUG { 23:42:58 DEBUG { 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:42:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:42:58 DEBUG }, 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:42:58 DEBUG "pod": "ds-idrepo-2" 23:42:58 DEBUG }, 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG "0" 23:42:58 DEBUG "120867101.81024523" 23:42:58 DEBUG ] 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713220978" 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG { 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "data": { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "result": [ 23:42:58 DEBUG { 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG "7.909819060592491" 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG { 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "data": { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "result": [ 23:42:58 DEBUG { 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG "0.082" 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG { 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "data": { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "result": [ 23:42:58 DEBUG { 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "pod": "ds-idrepo-2" 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "pod": "ds-idrepo-0" 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG "0" 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG { 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "data": { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "result": [ 23:42:58 DEBUG { 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG "1096.9666666666667" 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG } 23:42:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713220978" 23:42:58 INFO [http_cmd]: http status code OK 23:42:58 DEBUG --- status code --- 23:42:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:42:58 DEBUG --- http response --- 23:42:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:42:58 DEBUG ----- output ----- 23:42:58 DEBUG { 23:42:58 DEBUG "status": "success", 23:42:58 DEBUG "data": { 23:42:58 DEBUG "resultType": "vector", 23:42:58 DEBUG "result": [ 23:42:58 DEBUG { 23:42:58 DEBUG "metric": { 23:42:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG [...] 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG 23:42:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:42:58 DEBUG }, 23:42:58 DEBUG "value": [ 23:42:58 DEBUG 1713220978, 23:42:58 DEBUG "1903.8333333333333" 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG ] 23:42:58 DEBUG } 23:42:58 DEBUG } 23:43:00 INFO 23:43:00 INFO [loop_until]: kubectl --namespace=xlou top pods 23:43:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:43:00 INFO [loop_until]: OK (rc = 0) 23:43:00 DEBUG --- stdout --- 23:43:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 8226m 8640Mi am-7d597cf7d6-8vr8f 6814m 8911Mi am-7d597cf7d6-9wz9h 9789m 8985Mi ds-cts-0 12219m 8616Mi ds-cts-1 8694m 8679Mi ds-cts-2 7372m 8859Mi ds-idrepo-0 9338m 4989Mi ds-idrepo-1 1185m 2755Mi ds-idrepo-2 1101m 2763Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7970m 5080Mi 23:43:00 DEBUG --- stderr --- 23:43:00 DEBUG 23:43:00 INFO 23:43:00 INFO [loop_until]: kubectl --namespace=xlou top node 23:43:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:43:00 INFO [loop_until]: OK (rc = 0) 23:43:00 DEBUG --- stdout --- 23:43:00 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5072Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7894m 26% 10980Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8628m 28% 10715Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12790m 42% 3690Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1250m 4% 5645Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 12604m 42% 12910Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2972Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 7065m 23% 10420Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1216m 4% 5013Mi 4% gke-xlou-cdm-ds-0f601751-j488 8156m 27% 10552Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2447Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 11028m 36% 10667Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 8508m 28% 7174Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8158m 51% 8620Mi 14% 23:43:00 DEBUG --- stderr --- 23:43:00 DEBUG 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713221038" 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221038" 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221038" 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713221038" 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 ERROR [http_cmd]: ERROR 23:43:58 ERROR --- status code --- 23:43:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:43:58 ERROR --- http response --- 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221038" 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:43:58 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221038" 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG --- http response --- 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG { 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG "data": { 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG ----- output ----- 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG "result": [ 23:43:58 DEBUG { 23:43:58 DEBUG { 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG { 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG { 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG "metric": { 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG "data": { 23:43:58 DEBUG "data": { 23:43:58 DEBUG { 23:43:58 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:43:58 DEBUG "data": { 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:43:58 DEBUG --- http response --- 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:43:58 DEBUG { 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "status": "success", 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG }, 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "result": [ 23:43:58 DEBUG "result": [ 23:43:58 DEBUG "data": { 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG { 23:43:58 DEBUG { 23:43:58 DEBUG "result": [ 23:43:58 DEBUG { 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG "value": [ 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG "data": { 23:43:58 DEBUG { 23:43:58 DEBUG { 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG { 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG { 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG 23:43:58 DEBUG { 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "result": [ 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG "data": { 23:43:58 DEBUG "data": { 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "data": { 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "result": [ 23:43:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:43:58 DEBUG "instance": "10.142.0.107:9100" 23:43:58 DEBUG { 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "data": { 23:43:58 DEBUG 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG "data": { 23:43:58 DEBUG { 23:43:58 DEBUG }, 23:43:58 DEBUG }, 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "result": [ 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG "result": [ 23:43:58 DEBUG }, 23:43:58 DEBUG "result": [ 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG [...] 23:43:58 DEBUG { 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "instance": "10.142.0.107:9100" 23:43:58 DEBUG { 23:43:58 DEBUG { 23:43:58 DEBUG { 23:43:58 DEBUG "value": [ 23:43:58 DEBUG { 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG "result": [ 23:43:58 DEBUG { 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "result": [ 23:43:58 DEBUG "pod": "ds-idrepo-2" 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG }, 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "metric": { 23:43:58 DEBUG 23:43:58 DEBUG "metric": { 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG "metric": { 23:43:58 DEBUG 23:43:58 DEBUG "data": { 23:43:58 DEBUG { 23:43:58 DEBUG }, 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "pod": "ds-idrepo-2" 23:43:58 DEBUG "data": { 23:43:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:43:58 DEBUG 23:43:58 DEBUG "instance": "10.142.0.107:9100" 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:43:58 DEBUG 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG }, 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG }, 23:43:58 DEBUG 23:43:58 DEBUG }, 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "result": [ 23:43:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG [...] 23:43:58 DEBUG 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "result": [ 23:43:58 DEBUG "value": [ 23:43:58 DEBUG [...] 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:43:58 DEBUG { 23:43:58 DEBUG }, 23:43:58 DEBUG [...] 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG { 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG [...] 23:43:58 DEBUG 23:43:58 DEBUG [...] 23:43:58 DEBUG 23:43:58 DEBUG "metric": { 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG { 23:43:58 DEBUG "pod": "ds-idrepo-2" 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "instance": "10.142.0.107:9100" 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG }, 23:43:58 DEBUG [...] 23:43:58 DEBUG [...] 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG }, 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "instance": "10.142.0.43:9100" 23:43:58 DEBUG 23:43:58 DEBUG [...] 23:43:58 DEBUG }, 23:43:58 DEBUG [...] 23:43:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "data": { 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:43:58 DEBUG 23:43:58 DEBUG }, 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG }, 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG 23:43:58 DEBUG [...] 23:43:58 DEBUG }, 23:43:58 DEBUG "pod": "ds-idrepo-0" 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "instance": "10.142.0.43:9100" 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "0" 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "result": [ 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "value": [ 23:43:58 DEBUG }, 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG }, 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG ] 23:43:58 DEBUG "instance": "10.142.0.43:9100" 23:43:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:43:58 DEBUG { 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "6624.066666666667" 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "pod": "ds-idrepo-0" 23:43:58 DEBUG 23:43:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:43:58 DEBUG "6682378240" 23:43:58 DEBUG } 23:43:58 DEBUG }, 23:43:58 DEBUG }, 23:43:58 DEBUG "metric": { 23:43:58 DEBUG [...] 23:43:58 DEBUG 23:43:58 DEBUG "315153.5" 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG ] 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG }, 23:43:58 DEBUG [...] 23:43:58 DEBUG }, 23:43:58 DEBUG ] 23:43:58 DEBUG ] 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "pod": "ds-idrepo-2" 23:43:58 DEBUG 23:43:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:43:58 DEBUG ] 23:43:58 DEBUG "0" 23:43:58 DEBUG } 23:43:58 DEBUG "13921510400" 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG "value": [ 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG }, 23:43:58 DEBUG 23:43:58 DEBUG }, 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG ] 23:43:58 DEBUG ] 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG "12208.93333333333" 23:43:58 DEBUG "1364.1333333333332" 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG "value": [ 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG "0" 23:43:58 DEBUG 23:43:58 DEBUG "1364.1333333333332" 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG ] 23:43:58 DEBUG 23:43:58 DEBUG "pod": "ds-idrepo-0" 23:43:58 DEBUG }, 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG ] 23:43:58 DEBUG "instance": "10.142.0.43:9100" 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG 23:43:58 DEBUG } 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "0" 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG }, 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG ] 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 DEBUG 23:43:58 DEBUG ] 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221038" 23:43:58 DEBUG ] 23:43:58 DEBUG "value": [ 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG [...] 23:43:58 DEBUG } 23:43:58 DEBUG "0" 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG 1713221038, 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG "0" 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG } 23:43:58 DEBUG 23:43:58 DEBUG ] 23:43:58 DEBUG ] 23:43:58 DEBUG ] 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG "pod": "ds-cts-2" 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:43:58 DEBUG }, 23:43:58 DEBUG } 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG "value": [ 23:43:58 DEBUG } 23:43:58 DEBUG { 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "0" 23:43:58 DEBUG "data": { 23:43:58 DEBUG ] 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG } 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG "result": [ 23:43:58 DEBUG ] 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG { 23:43:58 DEBUG } 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG "metric": { 23:43:58 DEBUG } 23:43:58 DEBUG "pod": "ds-idrepo-2" 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG }, 23:43:58 DEBUG ----- output ----- 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG "value": [ 23:43:58 DEBUG { 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG 23:43:58 DEBUG "data": { 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG [...] 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG "result": [ 23:43:58 DEBUG 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG { 23:43:58 DEBUG { 23:43:58 DEBUG 23:43:58 DEBUG { 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "metric": { 23:43:58 DEBUG 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "data": { 23:43:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:43:58 DEBUG "pod": "ds-idrepo-0" 23:43:58 DEBUG "data": { 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG }, 23:43:58 DEBUG }, 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "result": [ 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "value": [ 23:43:58 DEBUG "result": [ 23:43:58 DEBUG { 23:43:58 DEBUG 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG { 23:43:58 DEBUG "metric": { 23:43:58 DEBUG 23:43:58 DEBUG "0" 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "pod": "ds-idrepo-2" 23:43:58 DEBUG 23:43:58 DEBUG ] 23:43:58 DEBUG "instance": "10.142.0.107:9100" 23:43:58 DEBUG }, 23:43:58 DEBUG [...] 23:43:58 DEBUG } 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG ] 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG } 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG } 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 23:43:58 DEBUG 23:43:58 DEBUG [...] 23:43:58 DEBUG }, 23:43:58 DEBUG [...] 23:43:58 DEBUG 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "0" 23:43:58 DEBUG 23:43:58 DEBUG "pod": "ds-idrepo-0" 23:43:58 DEBUG ] 23:43:58 DEBUG "instance": "10.142.0.43:9100" 23:43:58 DEBUG }, 23:43:58 DEBUG } 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG ] 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG } 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG "0" 23:43:58 DEBUG } 23:43:58 DEBUG "0.00030000000000001136" 23:43:58 DEBUG ] 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG { 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "data": { 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "result": [ 23:43:58 DEBUG { 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG [...] 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG "0" 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG { 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "data": { 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "result": [ 23:43:58 DEBUG { 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG [...] 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG "0" 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG { 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "data": { 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "result": [ 23:43:58 DEBUG { 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG [...] 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG "0.082" 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713221038" 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG { 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "data": { 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "result": [ 23:43:58 DEBUG { 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG [...] 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG "8.023760655682578" 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG { 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "data": { 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "result": [ 23:43:58 DEBUG { 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "pod": "ds-idrepo-2" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG [...] 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "pod": "ds-idrepo-0" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG "0" 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG { 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "data": { 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "result": [ 23:43:58 DEBUG { 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG [...] 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG "1364.1333333333334" 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG } 23:43:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221038" 23:43:58 INFO [http_cmd]: http status code OK 23:43:58 DEBUG --- status code --- 23:43:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:43:58 DEBUG --- http response --- 23:43:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:43:58 DEBUG ----- output ----- 23:43:58 DEBUG { 23:43:58 DEBUG "status": "success", 23:43:58 DEBUG "data": { 23:43:58 DEBUG "resultType": "vector", 23:43:58 DEBUG "result": [ 23:43:58 DEBUG { 23:43:58 DEBUG "metric": { 23:43:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG [...] 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG 23:43:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:43:58 DEBUG }, 23:43:58 DEBUG "value": [ 23:43:58 DEBUG 1713221038, 23:43:58 DEBUG "239.83333333333334" 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG ] 23:43:58 DEBUG } 23:43:58 DEBUG } 23:44:00 INFO 23:44:00 INFO [loop_until]: kubectl --namespace=xlou top pods 23:44:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:44:00 INFO [loop_until]: OK (rc = 0) 23:44:00 DEBUG --- stdout --- 23:44:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6255m 8639Mi am-7d597cf7d6-8vr8f 4758m 8660Mi am-7d597cf7d6-9wz9h 7672m 8975Mi ds-cts-0 4962m 8613Mi ds-cts-1 3187m 8679Mi ds-cts-2 3586m 8829Mi ds-idrepo-0 6994m 4989Mi ds-idrepo-1 1044m 2755Mi ds-idrepo-2 989m 2763Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8037m 7148Mi 23:44:00 DEBUG --- stderr --- 23:44:00 DEBUG 23:44:00 INFO 23:44:00 INFO [loop_until]: kubectl --namespace=xlou top node 23:44:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:44:00 INFO [loop_until]: OK (rc = 0) 23:44:00 DEBUG --- stdout --- 23:44:00 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5091Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3844m 12% 11002Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6641m 22% 10720Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11744m 39% 3653Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1079m 3% 5646Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 5532m 18% 12955Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2978Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 5238m 17% 10169Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1149m 3% 5015Mi 4% gke-xlou-cdm-ds-0f601751-j488 3599m 12% 10529Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2447Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 8997m 30% 10630Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 7539m 25% 7169Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8205m 51% 10449Mi 17% 23:44:00 DEBUG --- stderr --- 23:44:00 DEBUG 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713221098" 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713221098" 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221098" 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221098" 23:44:58 ERROR [http_cmd]: ERROR 23:44:58 ERROR --- status code --- 23:44:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:44:58 ERROR --- http response --- 23:44:58 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:44:58 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 DEBUG --- status code --- 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221098" 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG --- http response --- 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221098" 23:44:58 DEBUG --- status code --- 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG { 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG "data": { 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:44:58 DEBUG --- http response --- 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG { 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG { 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG ----- output ----- 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG { 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:44:58 DEBUG "result": [ 23:44:58 DEBUG { 23:44:58 DEBUG { 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG { 23:44:58 DEBUG { 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG "data": { 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG { 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "data": { 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG { 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG "data": { 23:44:58 DEBUG { 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG { 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "data": { 23:44:58 DEBUG "data": { 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG "data": { 23:44:58 DEBUG "data": { 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "result": [ 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG "result": [ 23:44:58 DEBUG "result": [ 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:44:58 DEBUG "data": { 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221098" 23:44:58 DEBUG "result": [ 23:44:58 DEBUG ----- output ----- 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 DEBUG "data": { 23:44:58 DEBUG { 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG "data": { 23:44:58 DEBUG }, 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG { 23:44:58 DEBUG { 23:44:58 DEBUG "result": [ 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG "result": [ 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG { 23:44:58 DEBUG { 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG "metric": { 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG "value": [ 23:44:58 DEBUG "result": [ 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "metric": { 23:44:58 DEBUG { 23:44:58 DEBUG { 23:44:58 DEBUG { 23:44:58 DEBUG { 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG "result": [ 23:44:58 DEBUG "pod": "ds-idrepo-2" 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG "result": [ 23:44:58 DEBUG { 23:44:58 DEBUG { 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "result": [ 23:44:58 DEBUG "data": { 23:44:58 DEBUG "pod": "ds-idrepo-2" 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 DEBUG "data": { 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG { 23:44:58 DEBUG }, 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "instance": "10.142.0.107:9100" 23:44:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:44:58 DEBUG }, 23:44:58 DEBUG "pod": "ds-idrepo-2" 23:44:58 DEBUG "data": { 23:44:58 DEBUG "instance": "10.142.0.107:9100" 23:44:58 DEBUG { 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG }, 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "value": [ 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:44:58 DEBUG 23:44:58 DEBUG "pod": "ds-idrepo-2" 23:44:58 DEBUG }, 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG }, 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG }, 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "result": [ 23:44:58 DEBUG "value": [ 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG "result": [ 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:44:58 DEBUG 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 23:44:58 DEBUG "value": [ 23:44:58 DEBUG "result": [ 23:44:58 DEBUG "value": [ 23:44:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG { 23:44:58 DEBUG 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG { 23:44:58 DEBUG { 23:44:58 DEBUG }, 23:44:58 DEBUG 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG { 23:44:58 DEBUG [...] 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG { 23:44:58 DEBUG 23:44:58 DEBUG }, 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG "metric": { 23:44:58 DEBUG 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 23:44:58 DEBUG { 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG "metric": { 23:44:58 DEBUG 23:44:58 DEBUG "value": [ 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:44:58 DEBUG 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG "instance": "10.142.0.107:9100" 23:44:58 DEBUG "data": { 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG [...] 23:44:58 DEBUG 23:44:58 DEBUG "data": { 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG [...] 23:44:58 DEBUG 23:44:58 DEBUG "instance": "10.142.0.107:9100" 23:44:58 DEBUG 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG }, 23:44:58 DEBUG [...] 23:44:58 DEBUG }, 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:44:58 DEBUG 23:44:58 DEBUG "result": [ 23:44:58 DEBUG 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG [...] 23:44:58 DEBUG [...] 23:44:58 DEBUG 23:44:58 DEBUG [...] 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 23:44:58 DEBUG "value": [ 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG 23:44:58 DEBUG { 23:44:58 DEBUG 23:44:58 DEBUG "data": { 23:44:58 DEBUG "result": [ 23:44:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:44:58 DEBUG [...] 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG [...] 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG { 23:44:58 DEBUG 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "pod": "ds-idrepo-0" 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG { 23:44:58 DEBUG }, 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG [...] 23:44:58 DEBUG { 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG [...] 23:44:58 DEBUG "pod": "ds-idrepo-2" 23:44:58 DEBUG }, 23:44:58 DEBUG "result": [ 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:44:58 DEBUG 23:44:58 DEBUG "data": { 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG "data": { 23:44:58 DEBUG 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG { 23:44:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG 23:44:58 DEBUG "instance": "10.142.0.43:9100" 23:44:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:44:58 DEBUG }, 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG "pod": "ds-idrepo-0" 23:44:58 DEBUG [...] 23:44:58 DEBUG 23:44:58 DEBUG "instance": "10.142.0.43:9100" 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG "pod": "ds-cts-2" 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG [...] 23:44:58 DEBUG [...] 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG "metric": { 23:44:58 DEBUG }, 23:44:58 DEBUG "746.3784110979884" 23:44:58 DEBUG "pod": "ds-idrepo-0" 23:44:58 DEBUG }, 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG }, 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG }, 23:44:58 DEBUG { 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "result": [ 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG "result": [ 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG "0" 23:44:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:44:58 DEBUG "value": [ 23:44:58 DEBUG ] 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG "8265519104" 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG "value": [ 23:44:58 DEBUG }, 23:44:58 DEBUG "data": { 23:44:58 DEBUG { 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "pod": "ds-idrepo-2" 23:44:58 DEBUG { 23:44:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:44:58 DEBUG 23:44:58 DEBUG ] 23:44:58 DEBUG }, 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG ] 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG 23:44:58 DEBUG "pod": "ds-idrepo-2" 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG }, 23:44:58 DEBUG "metric": { 23:44:58 DEBUG }, 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 23:44:58 DEBUG ] 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG "12208.533333333331" 23:44:58 DEBUG "1237" 23:44:58 DEBUG { 23:44:58 DEBUG } 23:44:58 DEBUG "0" 23:44:58 DEBUG "instance": "10.142.0.43:9100" 23:44:58 DEBUG }, 23:44:58 DEBUG "0" 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG "result": [ 23:44:58 DEBUG { 23:44:58 DEBUG 23:44:58 DEBUG "value": [ 23:44:58 DEBUG "instance": "10.142.0.107:9100" 23:44:58 DEBUG "value": [ 23:44:58 DEBUG [...] 23:44:58 DEBUG ] 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG "0" 23:44:58 DEBUG ] 23:44:58 DEBUG ] 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "data": { 23:44:58 DEBUG ] 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG ] 23:44:58 DEBUG "0" 23:44:58 DEBUG 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "instance": "10.142.0.43:9100" 23:44:58 DEBUG 23:44:58 DEBUG }, 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG 23:44:58 DEBUG [...] 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG } 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:44:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:44:58 DEBUG }, 23:44:58 DEBUG 23:44:58 DEBUG "value": [ 23:44:58 DEBUG "0" 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG "result": [ 23:44:58 DEBUG } 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG "0" 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG }, 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG ] 23:44:58 DEBUG 23:44:58 DEBUG [...] 23:44:58 DEBUG 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG { 23:44:58 DEBUG "5242.3" 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG "value": [ 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG [...] 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG "pod": "ds-idrepo-0" 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG "metric": { 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG 23:44:58 DEBUG "13922051072" 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG ] 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG "292157" 23:44:58 DEBUG 23:44:58 DEBUG ] 23:44:58 DEBUG 23:44:58 DEBUG [...] 23:44:58 DEBUG } 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:44:58 DEBUG 23:44:58 DEBUG }, 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG "0" 23:44:58 DEBUG }, 23:44:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:44:58 DEBUG "value": [ 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG [...] 23:44:58 DEBUG ] 23:44:58 DEBUG "pod": "ds-idrepo-0" 23:44:58 DEBUG 23:44:58 DEBUG ] 23:44:58 DEBUG "value": [ 23:44:58 DEBUG }, 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG }, 23:44:58 DEBUG 23:44:58 DEBUG "pod": "ds-idrepo-2" 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG "1238.3333333333333" 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG "value": [ 23:44:58 DEBUG "instance": "10.142.0.43:9100" 23:44:58 DEBUG }, 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG "0.082" 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG } 23:44:58 DEBUG [...] 23:44:58 DEBUG ] 23:44:58 DEBUG ] 23:44:58 DEBUG "0" 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG "0" 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG "0.00040000000000001514" 23:44:58 DEBUG ] 23:44:58 DEBUG 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG }, 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG "value": [ 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG } 23:44:58 DEBUG "0" 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713221098" 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG { 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "data": { 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG "result": [ 23:44:58 DEBUG { 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG [...] 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG "8.02621544628225" 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG { 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "data": { 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG "result": [ 23:44:58 DEBUG { 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "pod": "ds-idrepo-2" 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG [...] 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG "pod": "ds-idrepo-0" 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG "0" 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221098" 23:44:58 INFO [http_cmd]: http status code OK 23:44:58 DEBUG --- status code --- 23:44:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:58 DEBUG --- http response --- 23:44:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:44:58 DEBUG ----- output ----- 23:44:58 DEBUG { 23:44:58 DEBUG "status": "success", 23:44:58 DEBUG "data": { 23:44:58 DEBUG "resultType": "vector", 23:44:58 DEBUG "result": [ 23:44:58 DEBUG { 23:44:58 DEBUG "metric": { 23:44:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG [...] 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG 23:44:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:44:58 DEBUG }, 23:44:58 DEBUG "value": [ 23:44:58 DEBUG 1713221098, 23:44:58 DEBUG "1238.2666666666667" 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG ] 23:44:58 DEBUG } 23:44:58 DEBUG } 23:44:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221099" 23:44:59 INFO [http_cmd]: http status code OK 23:44:59 DEBUG --- status code --- 23:44:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:44:59 DEBUG --- http response --- 23:44:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:44:59 DEBUG ----- output ----- 23:44:59 DEBUG { 23:44:59 DEBUG "status": "success", 23:44:59 DEBUG "data": { 23:44:59 DEBUG "resultType": "vector", 23:44:59 DEBUG "result": [ 23:44:59 DEBUG { 23:44:59 DEBUG "metric": { 23:44:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:44:59 DEBUG }, 23:44:59 DEBUG "value": [ 23:44:59 DEBUG 23:44:59 DEBUG 23:44:59 DEBUG 23:44:59 DEBUG [...] 23:44:59 DEBUG 23:44:59 DEBUG 23:44:59 DEBUG 23:44:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:44:59 DEBUG }, 23:44:59 DEBUG "value": [ 23:44:59 DEBUG 1713221099, 23:44:59 DEBUG "2387.866666666667" 23:44:59 DEBUG ] 23:44:59 DEBUG } 23:44:59 DEBUG ] 23:44:59 DEBUG } 23:44:59 DEBUG } 23:45:00 INFO 23:45:00 INFO [loop_until]: kubectl --namespace=xlou top pods 23:45:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:45:00 INFO [loop_until]: OK (rc = 0) 23:45:00 DEBUG --- stdout --- 23:45:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 7954m 8639Mi am-7d597cf7d6-8vr8f 6223m 8662Mi am-7d597cf7d6-9wz9h 9689m 8976Mi ds-cts-0 11754m 8581Mi ds-cts-1 8537m 8689Mi ds-cts-2 8341m 8807Mi ds-idrepo-0 7588m 4989Mi ds-idrepo-1 1055m 2755Mi ds-idrepo-2 1099m 2763Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7853m 7910Mi 23:45:00 DEBUG --- stderr --- 23:45:00 DEBUG 23:45:00 INFO 23:45:00 INFO [loop_until]: kubectl --namespace=xlou top node 23:45:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:45:00 INFO [loop_until]: OK (rc = 0) 23:45:00 DEBUG --- stdout --- 23:45:00 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5077Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8375m 28% 10996Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8234m 27% 10714Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11037m 36% 3665Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1154m 3% 5636Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 12629m 42% 12933Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2978Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 6618m 22% 10186Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1179m 3% 5014Mi 4% gke-xlou-cdm-ds-0f601751-j488 8353m 27% 10505Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2449Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 10687m 35% 10648Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 8065m 27% 7174Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 7963m 50% 11267Mi 19% 23:45:00 DEBUG --- stderr --- 23:45:00 DEBUG 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713221158" 23:45:58 ERROR [http_cmd]: ERROR 23:45:58 ERROR --- status code --- 23:45:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:45:58 ERROR --- http response --- 23:45:58 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:45:58 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG --- http response --- 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG { 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "data": { 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG "result": [ 23:45:58 DEBUG { 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "pod": "ds-cts-1" 23:45:58 DEBUG }, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG [...] 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:45:58 DEBUG }, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG "0" 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG { 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "data": { 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG "result": [ 23:45:58 DEBUG { 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "pod": "ds-cts-1" 23:45:58 DEBUG }, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG [...] 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:45:58 DEBUG }, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG "0" 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG { 23:45:58 DEBUG { 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "data": { 23:45:58 DEBUG "data": { 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG "result": [ 23:45:58 DEBUG "result": [ 23:45:58 DEBUG { 23:45:58 DEBUG { 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:45:58 DEBUG "instance": "10.142.0.107:9100" 23:45:58 DEBUG }, 23:45:58 DEBUG }, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "value": [ 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG --- http response --- 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG [...] 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG [...] 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG { 23:45:58 DEBUG { 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:45:58 DEBUG "instance": "10.142.0.43:9100" 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:45:58 DEBUG "data": { 23:45:58 DEBUG "data": { 23:45:58 DEBUG }, 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG }, 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG "value": [ 23:45:58 DEBUG { 23:45:58 DEBUG "value": [ 23:45:58 DEBUG { 23:45:58 DEBUG { 23:45:58 DEBUG "result": [ 23:45:58 DEBUG "result": [ 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG { 23:45:58 DEBUG { 23:45:58 DEBUG "8478466048" 23:45:58 DEBUG "data": { 23:45:58 DEBUG "5311.266666666666" 23:45:58 DEBUG "data": { 23:45:58 DEBUG "data": { 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "metric": { 23:45:58 DEBUG ] 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG ] 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG "instance": "10.142.0.107:9100" 23:45:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:45:58 DEBUG } 23:45:58 DEBUG "result": [ 23:45:58 DEBUG } 23:45:58 DEBUG "result": [ 23:45:58 DEBUG "result": [ 23:45:58 DEBUG }, 23:45:58 DEBUG }, 23:45:58 DEBUG ] 23:45:58 DEBUG { 23:45:58 DEBUG ] 23:45:58 DEBUG { 23:45:58 DEBUG { 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "value": [ 23:45:58 DEBUG } 23:45:58 DEBUG "metric": { 23:45:58 DEBUG } 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "metric": { 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG } 23:45:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:45:58 DEBUG } 23:45:58 DEBUG "pod": "ds-idrepo-2" 23:45:58 DEBUG "instance": "10.142.0.107:9100" 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG }, 23:45:58 DEBUG }, 23:45:58 DEBUG }, 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "value": [ 23:45:58 DEBUG [...] 23:45:58 DEBUG [...] 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG [...] 23:45:58 DEBUG [...] 23:45:58 DEBUG [...] 23:45:58 DEBUG "instance": "10.142.0.43:9100" 23:45:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG }, 23:45:58 DEBUG }, 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "value": [ 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:45:58 DEBUG "pod": "ds-cts-2" 23:45:58 DEBUG "instance": "10.142.0.43:9100" 23:45:58 DEBUG "0.0003666666666669964" 23:45:58 DEBUG "1196.0666666666666" 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG }, 23:45:58 DEBUG }, 23:45:58 DEBUG }, 23:45:58 DEBUG ] 23:45:58 DEBUG ] 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "value": [ 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG ] 23:45:58 DEBUG ] 23:45:58 DEBUG "234432" 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG "0" 23:45:58 DEBUG "13922708480" 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG { 23:45:58 DEBUG ] 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG "data": { 23:45:58 DEBUG ] 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG } 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG "result": [ 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 DEBUG { 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "pod": "ds-cts-1" 23:45:58 DEBUG }, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG [...] 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:45:58 DEBUG }, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG "0" 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG --- status code --- 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG --- status code --- 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG --- http response --- 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG --- status code --- 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG ----- output ----- 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG { 23:45:58 DEBUG --- http response --- 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG { 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG "status": "success", 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:45:58 DEBUG "data": { 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG "data": { 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG { 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG { 23:45:58 DEBUG { 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:45:58 DEBUG { 23:45:58 DEBUG "result": [ 23:45:58 DEBUG { 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:45:58 DEBUG "result": [ 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG "data": { 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG { 23:45:58 DEBUG "status": "success", 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "data": { 23:45:58 DEBUG "data": { 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG { 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:45:58 DEBUG { 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "data": { 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG "data": { 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG { 23:45:58 DEBUG "metric": { 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG "result": [ 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "data": { 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG "result": [ 23:45:58 DEBUG "result": [ 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "pod": "ds-idrepo-2" 23:45:58 DEBUG { 23:45:58 DEBUG { 23:45:58 DEBUG { 23:45:58 DEBUG "pod": "ds-idrepo-2" 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG "result": [ 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG "result": [ 23:45:58 DEBUG { 23:45:58 DEBUG { 23:45:58 DEBUG "data": { 23:45:58 DEBUG }, 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG }, 23:45:58 DEBUG "result": [ 23:45:58 DEBUG { 23:45:58 DEBUG { 23:45:58 DEBUG "metric": { 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "data": { 23:45:58 DEBUG "pod": "ds-idrepo-2" 23:45:58 DEBUG "data": { 23:45:58 DEBUG "value": [ 23:45:58 DEBUG { 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "pod": "ds-cts-1" 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:45:58 DEBUG "result": [ 23:45:58 DEBUG 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG }, 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "pod": "ds-idrepo-2" 23:45:58 DEBUG "instance": "10.142.0.107:9100" 23:45:58 DEBUG }, 23:45:58 DEBUG { 23:45:58 DEBUG }, 23:45:58 DEBUG { 23:45:58 DEBUG 23:45:58 DEBUG "result": [ 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "result": [ 23:45:58 DEBUG 23:45:58 DEBUG "pod": "ds-idrepo-2" 23:45:58 DEBUG }, 23:45:58 DEBUG }, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "metric": { 23:45:58 DEBUG 23:45:58 DEBUG { 23:45:58 DEBUG 23:45:58 DEBUG { 23:45:58 DEBUG 23:45:58 DEBUG }, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG "data": { 23:45:58 DEBUG 23:45:58 DEBUG "instance": "10.142.0.107:9100" 23:45:58 DEBUG [...] 23:45:58 DEBUG "metric": { 23:45:58 DEBUG 23:45:58 DEBUG "metric": { 23:45:58 DEBUG [...] 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG 23:45:58 DEBUG }, 23:45:58 DEBUG 23:45:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:45:58 DEBUG 23:45:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG "result": [ 23:45:58 DEBUG 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG }, 23:45:58 DEBUG [...] 23:45:58 DEBUG }, 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG [...] 23:45:58 DEBUG { 23:45:58 DEBUG [...] 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG [...] 23:45:58 DEBUG [...] 23:45:58 DEBUG 23:45:58 DEBUG "metric": { 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG "pod": "ds-idrepo-0" 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG "pod": "ds-idrepo-0" 23:45:58 DEBUG [...] 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG }, 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG }, 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG }, 23:45:58 DEBUG 23:45:58 DEBUG [...] 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG "pod": "ds-idrepo-0" 23:45:58 DEBUG 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:45:58 DEBUG 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG [...] 23:45:58 DEBUG }, 23:45:58 DEBUG [...] 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG 23:45:58 DEBUG "pod": "ds-idrepo-0" 23:45:58 DEBUG "instance": "10.142.0.43:9100" 23:45:58 DEBUG }, 23:45:58 DEBUG 23:45:58 DEBUG }, 23:45:58 DEBUG 23:45:58 DEBUG "0" 23:45:58 DEBUG 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG "0" 23:45:58 DEBUG "pod": "ds-idrepo-0" 23:45:58 DEBUG }, 23:45:58 DEBUG }, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG ] 23:45:58 DEBUG 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG 23:45:58 DEBUG ] 23:45:58 DEBUG }, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG "instance": "10.142.0.43:9100" 23:45:58 DEBUG } 23:45:58 DEBUG 23:45:58 DEBUG "0" 23:45:58 DEBUG 23:45:58 DEBUG } 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG "0" 23:45:58 DEBUG [...] 23:45:58 DEBUG "8.031659929342633" 23:45:58 DEBUG }, 23:45:58 DEBUG ] 23:45:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:45:58 DEBUG }, 23:45:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:45:58 DEBUG ] 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG "0" 23:45:58 DEBUG "12287.83333333333" 23:45:58 DEBUG ] 23:45:58 DEBUG 23:45:58 DEBUG ] 23:45:58 DEBUG "value": [ 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG "value": [ 23:45:58 DEBUG }, 23:45:58 DEBUG } 23:45:58 DEBUG "0" 23:45:58 DEBUG ] 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG 23:45:58 DEBUG } 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG 23:45:58 DEBUG ] 23:45:58 DEBUG "0" 23:45:58 DEBUG ] 23:45:58 DEBUG "1196.1333333333332" 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG "0.082" 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG }, 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG "value": [ 23:45:58 DEBUG ] 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG "0" 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221158" 23:45:58 INFO [http_cmd]: http status code OK 23:45:58 DEBUG --- status code --- 23:45:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:58 DEBUG --- http response --- 23:45:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:45:58 DEBUG ----- output ----- 23:45:58 DEBUG { 23:45:58 DEBUG "status": "success", 23:45:58 DEBUG "data": { 23:45:58 DEBUG "resultType": "vector", 23:45:58 DEBUG "result": [ 23:45:58 DEBUG { 23:45:58 DEBUG "metric": { 23:45:58 DEBUG "pod": "ds-idrepo-2" 23:45:58 DEBUG }, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG [...] 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG 23:45:58 DEBUG "pod": "ds-idrepo-0" 23:45:58 DEBUG }, 23:45:58 DEBUG "value": [ 23:45:58 DEBUG 1713221158, 23:45:58 DEBUG "0" 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG ] 23:45:58 DEBUG } 23:45:58 DEBUG } 23:45:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221159" 23:45:59 INFO [http_cmd]: http status code OK 23:45:59 DEBUG --- status code --- 23:45:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:59 DEBUG --- http response --- 23:45:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:45:59 DEBUG ----- output ----- 23:45:59 DEBUG { 23:45:59 DEBUG "status": "success", 23:45:59 DEBUG "data": { 23:45:59 DEBUG "resultType": "vector", 23:45:59 DEBUG "result": [ 23:45:59 DEBUG { 23:45:59 DEBUG "metric": { 23:45:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:45:59 DEBUG }, 23:45:59 DEBUG "value": [ 23:45:59 DEBUG 23:45:59 DEBUG 23:45:59 DEBUG 23:45:59 DEBUG [...] 23:45:59 DEBUG 23:45:59 DEBUG 23:45:59 DEBUG 23:45:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:45:59 DEBUG }, 23:45:59 DEBUG "value": [ 23:45:59 DEBUG 1713221159, 23:45:59 DEBUG "1559.5666666666666" 23:45:59 DEBUG ] 23:45:59 DEBUG } 23:45:59 DEBUG ] 23:45:59 DEBUG } 23:45:59 DEBUG } 23:45:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221159" 23:45:59 INFO [http_cmd]: http status code OK 23:45:59 DEBUG --- status code --- 23:45:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:45:59 DEBUG --- http response --- 23:45:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:45:59 DEBUG ----- output ----- 23:45:59 DEBUG { 23:45:59 DEBUG "status": "success", 23:45:59 DEBUG "data": { 23:45:59 DEBUG "resultType": "vector", 23:45:59 DEBUG "result": [ 23:45:59 DEBUG { 23:45:59 DEBUG "metric": { 23:45:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:45:59 DEBUG }, 23:45:59 DEBUG "value": [ 23:45:59 DEBUG 23:45:59 DEBUG 23:45:59 DEBUG 23:45:59 DEBUG [...] 23:45:59 DEBUG 23:45:59 DEBUG 23:45:59 DEBUG 23:45:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:45:59 DEBUG }, 23:45:59 DEBUG "value": [ 23:45:59 DEBUG 1713221159, 23:45:59 DEBUG "1196.1333333333334" 23:45:59 DEBUG ] 23:45:59 DEBUG } 23:45:59 DEBUG ] 23:45:59 DEBUG } 23:45:59 DEBUG } 23:46:00 INFO 23:46:00 INFO [loop_until]: kubectl --namespace=xlou top pods 23:46:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:46:00 INFO [loop_until]: OK (rc = 0) 23:46:00 DEBUG --- stdout --- 23:46:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6962m 8640Mi am-7d597cf7d6-8vr8f 5351m 8661Mi am-7d597cf7d6-9wz9h 8148m 8984Mi ds-cts-0 8602m 8585Mi ds-cts-1 5553m 8653Mi ds-cts-2 6022m 8805Mi ds-idrepo-0 7017m 4989Mi ds-idrepo-1 960m 2755Mi ds-idrepo-2 967m 2763Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1026Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8085m 8088Mi 23:46:00 DEBUG --- stderr --- 23:46:00 DEBUG 23:46:01 INFO 23:46:01 INFO [loop_until]: kubectl --namespace=xlou top node 23:46:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:46:01 INFO [loop_until]: OK (rc = 0) 23:46:01 DEBUG --- stdout --- 23:46:01 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5069Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5381m 18% 10994Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6899m 23% 10716Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10341m 34% 3643Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1037m 3% 5650Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 7952m 26% 12933Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2976Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 5454m 18% 10173Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 979m 3% 5022Mi 4% gke-xlou-cdm-ds-0f601751-j488 5446m 18% 10506Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2445Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 7822m 26% 10649Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 6524m 21% 7177Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8344m 52% 10989Mi 18% 23:46:01 DEBUG --- stderr --- 23:46:01 DEBUG 23:46:16 WARNING File /tmp/lodemon_data-20240415_230746/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [15/Apr/2024 23:46:16] "GET /monitoring/average?start_time=20240415_224215&stop_time=20240415_224415 HTTP/1.1" 200 - 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 ERROR [http_cmd]: ERROR 23:46:58 ERROR --- status code --- 23:46:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:46:58 ERROR --- http response --- 23:46:58 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:46:58 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221218" 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG { 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG "data": { 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "result": [ 23:46:58 DEBUG { 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG "170069" 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221218" 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG { 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG { 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG { 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG "data": { 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG "data": { 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "data": { 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "result": [ 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "result": [ 23:46:58 DEBUG { 23:46:58 DEBUG "result": [ 23:46:58 DEBUG { 23:46:58 DEBUG "metric": { 23:46:58 DEBUG { 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "pod": "ds-idrepo-2" 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "instance": "10.142.0.107:9100" 23:46:58 DEBUG }, 23:46:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "pod": "ds-cts-2" 23:46:58 DEBUG 23:46:58 DEBUG "instance": "10.142.0.43:9100" 23:46:58 DEBUG }, 23:46:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG "0" 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG "5446.599999999999" 23:46:58 DEBUG ] 23:46:58 DEBUG "0.0029760239446604267" 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG { 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG "data": { 23:46:58 DEBUG { 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG "result": [ 23:46:58 DEBUG "data": { 23:46:58 DEBUG { 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "result": [ 23:46:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:46:58 DEBUG { 23:46:58 DEBUG }, 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "value": [ 23:46:58 DEBUG "instance": "10.142.0.107:9100" 23:46:58 DEBUG 23:46:58 DEBUG }, 23:46:58 DEBUG 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:46:58 DEBUG 23:46:58 DEBUG }, 23:46:58 DEBUG 23:46:58 DEBUG "value": [ 23:46:58 DEBUG "instance": "10.142.0.43:9100" 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG }, 23:46:58 DEBUG "0" 23:46:58 DEBUG "value": [ 23:46:58 DEBUG ] 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG } 23:46:58 DEBUG "0.0004666666666665264" 23:46:58 DEBUG ] 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG { 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG "data": { 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "result": [ 23:46:58 DEBUG { 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG "838.4839817139913" 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221218" 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG { 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG "data": { 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "result": [ 23:46:58 DEBUG { 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG "113942528" 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG { 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG "data": { 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "result": [ 23:46:58 DEBUG { 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG "0.082" 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG { 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG "data": { 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "result": [ 23:46:58 DEBUG { 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "instance": "10.142.0.107:9100" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "instance": "10.142.0.43:9100" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG "13923363840" 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG { 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG "data": { 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "result": [ 23:46:58 DEBUG { 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG "73052.9169068315" 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221218" 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG --- status code --- 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG --- status code --- 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- http response --- 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:46:58 DEBUG --- http response --- 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:46:58 DEBUG --- http response --- 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG { 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG { 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG { 23:46:58 DEBUG { 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG { 23:46:58 DEBUG --- http response --- 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG { 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG "data": { 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG "data": { 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG "data": { 23:46:58 DEBUG "data": { 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "data": { 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG { 23:46:58 DEBUG "data": { 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "result": [ 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG { 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG "result": [ 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG { 23:46:58 DEBUG "result": [ 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG "result": [ 23:46:58 DEBUG { 23:46:58 DEBUG "result": [ 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG { 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:46:58 DEBUG "data": { 23:46:58 INFO [http_cmd]: http status code OK 23:46:58 DEBUG "result": [ 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG { 23:46:58 DEBUG { 23:46:58 DEBUG { 23:46:58 DEBUG "metric": { 23:46:58 DEBUG { 23:46:58 DEBUG "data": { 23:46:58 DEBUG "metric": { 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG --- status code --- 23:46:58 DEBUG { 23:46:58 DEBUG "data": { 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "pod": "ds-idrepo-2" 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "pod": "ds-idrepo-2" 23:46:58 DEBUG { 23:46:58 DEBUG "result": [ 23:46:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:58 DEBUG "metric": { 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "pod": "ds-idrepo-2" 23:46:58 DEBUG "data": { 23:46:58 DEBUG "pod": "ds-idrepo-2" 23:46:58 DEBUG }, 23:46:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:46:58 DEBUG "result": [ 23:46:58 DEBUG }, 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG { 23:46:58 DEBUG --- http response --- 23:46:58 DEBUG "pod": "ds-idrepo-2" 23:46:58 DEBUG "result": [ 23:46:58 DEBUG }, 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "result": [ 23:46:58 DEBUG "value": [ 23:46:58 DEBUG }, 23:46:58 DEBUG { 23:46:58 DEBUG "value": [ 23:46:58 DEBUG "data": { 23:46:58 DEBUG "metric": { 23:46:58 DEBUG }, 23:46:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:46:58 DEBUG { 23:46:58 DEBUG "value": [ 23:46:58 DEBUG }, 23:46:58 DEBUG { 23:46:58 DEBUG 23:46:58 DEBUG "value": [ 23:46:58 DEBUG "metric": { 23:46:58 DEBUG 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG "pod": "ds-idrepo-2" 23:46:58 DEBUG "value": [ 23:46:58 DEBUG ----- output ----- 23:46:58 DEBUG "metric": { 23:46:58 DEBUG 23:46:58 DEBUG "value": [ 23:46:58 DEBUG "metric": { 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "instance": "10.142.0.107:9100" 23:46:58 DEBUG 23:46:58 DEBUG "result": [ 23:46:58 DEBUG }, 23:46:58 DEBUG 23:46:58 DEBUG { 23:46:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "instance": "10.142.0.107:9100" 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG }, 23:46:58 DEBUG 23:46:58 DEBUG { 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG "status": "success", 23:46:58 DEBUG }, 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG }, 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG "value": [ 23:46:58 DEBUG [...] 23:46:58 DEBUG "metric": { 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "data": { 23:46:58 DEBUG "value": [ 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "resultType": "vector", 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG }, 23:46:58 DEBUG [...] 23:46:58 DEBUG [...] 23:46:58 DEBUG "result": [ 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG { 23:46:58 DEBUG 23:46:58 DEBUG "pod": "ds-idrepo-0" 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG 23:46:58 DEBUG [...] 23:46:58 DEBUG "pod": "ds-idrepo-0" 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "metric": { 23:46:58 DEBUG [...] 23:46:58 DEBUG }, 23:46:58 DEBUG 23:46:58 DEBUG "pod": "ds-idrepo-0" 23:46:58 DEBUG 23:46:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:46:58 DEBUG 23:46:58 DEBUG }, 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG 23:46:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:46:58 DEBUG 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG }, 23:46:58 DEBUG 23:46:58 DEBUG }, 23:46:58 DEBUG 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG "pod": "ds-idrepo-0" 23:46:58 DEBUG "pod": "ds-idrepo-0" 23:46:58 DEBUG }, 23:46:58 DEBUG 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG "pod": "ds-idrepo-0" 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG [...] 23:46:58 DEBUG }, 23:46:58 DEBUG }, 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG "0" 23:46:58 DEBUG }, 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG "instance": "10.142.0.43:9100" 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG "instance": "10.142.0.43:9100" 23:46:58 DEBUG "0" 23:46:58 DEBUG 23:46:58 DEBUG "value": [ 23:46:58 DEBUG "value": [ 23:46:58 DEBUG 23:46:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:46:58 DEBUG ] 23:46:58 DEBUG "value": [ 23:46:58 DEBUG "0" 23:46:58 DEBUG }, 23:46:58 DEBUG "0" 23:46:58 DEBUG }, 23:46:58 DEBUG ] 23:46:58 DEBUG 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG 23:46:58 DEBUG }, 23:46:58 DEBUG } 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG ] 23:46:58 DEBUG "value": [ 23:46:58 DEBUG ] 23:46:58 DEBUG "value": [ 23:46:58 DEBUG } 23:46:58 DEBUG 23:46:58 DEBUG "0" 23:46:58 DEBUG "0" 23:46:58 DEBUG 23:46:58 DEBUG "value": [ 23:46:58 DEBUG ] 23:46:58 DEBUG "0" 23:46:58 DEBUG } 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG } 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG ] 23:46:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:46:58 DEBUG ] 23:46:58 DEBUG ] 23:46:58 DEBUG [...] 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG ] 23:46:58 DEBUG "0.0003333333333333262" 23:46:58 DEBUG ] 23:46:58 DEBUG "12378.299999999997" 23:46:58 DEBUG } 23:46:58 DEBUG }, 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG 23:46:58 DEBUG "0.034308480150128516" 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG "value": [ 23:46:58 DEBUG ] 23:46:58 DEBUG ] 23:46:58 DEBUG 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG ] 23:46:58 DEBUG "0" 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG }, 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:58 DEBUG "value": [ 23:46:58 DEBUG ] 23:46:58 DEBUG 1713221218, 23:46:58 DEBUG } 23:46:58 DEBUG "0" 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG ] 23:46:58 DEBUG } 23:46:58 DEBUG } 23:46:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221219" 23:46:59 INFO [http_cmd]: http status code OK 23:46:59 DEBUG --- status code --- 23:46:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:59 DEBUG --- http response --- 23:46:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:46:59 DEBUG ----- output ----- 23:46:59 DEBUG { 23:46:59 DEBUG "status": "success", 23:46:59 DEBUG "data": { 23:46:59 DEBUG "resultType": "vector", 23:46:59 DEBUG "result": [ 23:46:59 DEBUG { 23:46:59 DEBUG "metric": { 23:46:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:46:59 DEBUG }, 23:46:59 DEBUG "value": [ 23:46:59 DEBUG 23:46:59 DEBUG 23:46:59 DEBUG 23:46:59 DEBUG [...] 23:46:59 DEBUG 23:46:59 DEBUG 23:46:59 DEBUG 23:46:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:46:59 DEBUG }, 23:46:59 DEBUG "value": [ 23:46:59 DEBUG 1713221219, 23:46:59 DEBUG "1364.3333333333333" 23:46:59 DEBUG ] 23:46:59 DEBUG } 23:46:59 DEBUG ] 23:46:59 DEBUG } 23:46:59 DEBUG } 23:46:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221219" 23:46:59 INFO [http_cmd]: http status code OK 23:46:59 DEBUG --- status code --- 23:46:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:46:59 DEBUG --- http response --- 23:46:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:46:59 DEBUG ----- output ----- 23:46:59 DEBUG { 23:46:59 DEBUG "status": "success", 23:46:59 DEBUG "data": { 23:46:59 DEBUG "resultType": "vector", 23:46:59 DEBUG "result": [ 23:46:59 DEBUG { 23:46:59 DEBUG "metric": { 23:46:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:46:59 DEBUG }, 23:46:59 DEBUG "value": [ 23:46:59 DEBUG 23:46:59 DEBUG 23:46:59 DEBUG 23:46:59 DEBUG [...] 23:46:59 DEBUG 23:46:59 DEBUG 23:46:59 DEBUG 23:46:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:46:59 DEBUG }, 23:46:59 DEBUG "value": [ 23:46:59 DEBUG 1713221219, 23:46:59 DEBUG "0" 23:46:59 DEBUG ] 23:46:59 DEBUG } 23:46:59 DEBUG ] 23:46:59 DEBUG } 23:46:59 DEBUG } 23:47:00 INFO 23:47:00 INFO [loop_until]: kubectl --namespace=xlou top pods 23:47:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:47:00 INFO [loop_until]: OK (rc = 0) 23:47:00 DEBUG --- stdout --- 23:47:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 1279m 8591Mi am-7d597cf7d6-8vr8f 1113m 8662Mi am-7d597cf7d6-9wz9h 1595m 8977Mi ds-cts-0 4085m 8557Mi ds-cts-1 2916m 8668Mi ds-cts-2 3073m 8796Mi ds-idrepo-0 912m 4992Mi ds-idrepo-1 107m 2755Mi ds-idrepo-2 113m 2763Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 11m 1027Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 237m 290Mi 23:47:00 DEBUG --- stderr --- 23:47:00 DEBUG 23:47:01 INFO 23:47:01 INFO [loop_until]: kubectl --namespace=xlou top node 23:47:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:47:01 INFO [loop_until]: OK (rc = 0) 23:47:01 DEBUG --- stdout --- 23:47:01 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 5070Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2974m 9% 10985Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1332m 4% 10722Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 99m 0% 3115Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 169m 0% 5665Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 4622m 15% 12943Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 2979Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1199m 4% 10193Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 220m 0% 5075Mi 4% gke-xlou-cdm-ds-0f601751-j488 3011m 10% 10509Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 100m 0% 2454Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 1796m 6% 10612Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 1051m 3% 7248Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 73m 0% 3925Mi 6% 23:47:01 DEBUG --- stderr --- 23:47:01 DEBUG 23:47:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713221278" 23:47:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713221278" 23:47:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221278" 23:47:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221278" 23:47:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713221278" 23:47:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221278" 23:47:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221278" 23:47:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221278" 23:47:58 ERROR [http_cmd]: ERROR 23:47:58 INFO [http_cmd]: http status code OK 23:47:58 INFO [http_cmd]: http status code OK 23:47:58 INFO [http_cmd]: http status code OK 23:47:58 ERROR --- status code --- 23:47:58 DEBUG --- status code --- 23:47:58 DEBUG --- status code --- 23:47:58 DEBUG --- status code --- 23:47:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:47:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:58 ERROR --- http response --- 23:47:58 DEBUG --- http response --- 23:47:58 DEBUG --- http response --- 23:47:58 DEBUG --- http response --- 23:47:58 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:47:58 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:47:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221278" 23:47:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:47:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:47:58 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:47:58 DEBUG ----- output ----- 23:47:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221278" 23:47:58 INFO [http_cmd]: http status code OK 23:47:58 DEBUG ----- output ----- 23:47:58 INFO [http_cmd]: http status code OK 23:47:58 DEBUG { 23:47:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221278" 23:47:58 DEBUG --- status code --- 23:47:58 DEBUG { 23:47:58 DEBUG --- status code --- 23:47:58 DEBUG "status": "success", 23:47:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221278" 23:47:58 DEBUG "status": "success", 23:47:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:58 DEBUG --- http response --- 23:47:58 DEBUG --- http response --- 23:47:58 DEBUG "data": { 23:47:58 DEBUG "data": { 23:47:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:47:58 DEBUG "resultType": "vector", 23:47:58 INFO [http_cmd]: http status code OK 23:47:58 INFO [http_cmd]: http status code OK 23:47:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:47:58 DEBUG "resultType": "vector", 23:47:58 DEBUG ----- output ----- 23:47:58 DEBUG "result": [ 23:47:58 DEBUG --- status code --- 23:47:58 DEBUG --- status code --- 23:47:58 DEBUG ----- output ----- 23:47:58 DEBUG "result": [ 23:47:58 DEBUG { 23:47:58 DEBUG { 23:47:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:58 DEBUG { 23:47:58 DEBUG { 23:47:58 DEBUG "status": "success", 23:47:58 DEBUG "metric": { 23:47:58 DEBUG --- http response --- 23:47:58 DEBUG --- http response --- 23:47:58 DEBUG "status": "success", 23:47:58 DEBUG "metric": { 23:47:58 DEBUG "data": { 23:47:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:47:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:47:58 DEBUG "data": { 23:47:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:47:58 DEBUG "pod": "ds-idrepo-2" 23:47:58 DEBUG "resultType": "vector", 23:47:58 DEBUG }, 23:47:58 DEBUG ----- output ----- 23:47:58 DEBUG "resultType": "vector", 23:47:58 DEBUG ----- output ----- 23:47:58 DEBUG }, 23:47:58 DEBUG "result": [ 23:47:58 DEBUG "value": [ 23:47:58 DEBUG { 23:47:58 DEBUG "result": [ 23:47:58 DEBUG { 23:47:58 DEBUG "value": [ 23:47:58 DEBUG { 23:47:58 DEBUG 23:47:58 DEBUG "status": "success", 23:47:58 DEBUG { 23:47:58 DEBUG "status": "success", 23:47:58 DEBUG 23:47:58 DEBUG "metric": { 23:47:58 DEBUG 23:47:58 DEBUG "data": { 23:47:58 DEBUG "metric": { 23:47:58 DEBUG "data": { 23:47:58 DEBUG 23:47:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:47:58 DEBUG 23:47:58 DEBUG "resultType": "vector", 23:47:58 DEBUG "instance": "10.142.0.107:9100" 23:47:58 DEBUG "resultType": "vector", 23:47:58 DEBUG 23:47:58 DEBUG }, 23:47:58 DEBUG [...] 23:47:58 DEBUG "result": [ 23:47:58 DEBUG }, 23:47:58 DEBUG "result": [ 23:47:58 DEBUG [...] 23:47:58 DEBUG "value": [ 23:47:58 DEBUG 23:47:58 DEBUG { 23:47:58 DEBUG "value": [ 23:47:58 DEBUG { 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG "metric": { 23:47:58 DEBUG 23:47:58 DEBUG "metric": { 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:47:58 DEBUG 23:47:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:47:58 DEBUG }, 23:47:58 DEBUG 23:47:58 DEBUG }, 23:47:58 DEBUG "pod": "ds-cts-2" 23:47:58 DEBUG [...] 23:47:58 DEBUG }, 23:47:58 DEBUG "value": [ 23:47:58 DEBUG [...] 23:47:58 DEBUG "value": [ 23:47:58 DEBUG }, 23:47:58 DEBUG 23:47:58 DEBUG "value": [ 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG "value": [ 23:47:58 DEBUG 23:47:58 DEBUG 1713221278, 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 1713221278, 23:47:58 DEBUG 23:47:58 DEBUG "0" 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG "0" 23:47:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:47:58 DEBUG ] 23:47:58 DEBUG [...] 23:47:58 DEBUG "instance": "10.142.0.43:9100" 23:47:58 DEBUG [...] 23:47:58 DEBUG ] 23:47:58 DEBUG }, 23:47:58 DEBUG } 23:47:58 DEBUG 23:47:58 DEBUG }, 23:47:58 DEBUG 23:47:58 DEBUG } 23:47:58 DEBUG "value": [ 23:47:58 DEBUG ] 23:47:58 DEBUG 23:47:58 DEBUG "value": [ 23:47:58 DEBUG 23:47:58 DEBUG ] 23:47:58 DEBUG 1713221278, 23:47:58 DEBUG } 23:47:58 DEBUG 23:47:58 DEBUG 1713221278, 23:47:58 DEBUG 23:47:58 DEBUG } 23:47:58 INFO [http_cmd]: http status code OK 23:47:58 DEBUG "654487552" 23:47:58 DEBUG } 23:47:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:47:58 DEBUG "0.000500000000000019" 23:47:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:47:58 DEBUG } 23:47:58 DEBUG --- status code --- 23:47:58 DEBUG ] 23:47:58 DEBUG }, 23:47:58 DEBUG ] 23:47:58 DEBUG }, 23:47:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:58 DEBUG } 23:47:58 DEBUG "value": [ 23:47:58 DEBUG } 23:47:58 INFO [http_cmd]: http status code OK 23:47:58 DEBUG "value": [ 23:47:58 DEBUG --- http response --- 23:47:58 DEBUG ] 23:47:58 DEBUG 1713221278, 23:47:58 DEBUG ] 23:47:58 DEBUG --- status code --- 23:47:58 DEBUG 1713221278, 23:47:58 DEBUG } 23:47:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 23:47:58 DEBUG "92424.5" 23:47:58 INFO [http_cmd]: http status code OK 23:47:58 INFO [http_cmd]: http status code OK 23:47:58 DEBUG } 23:47:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:58 DEBUG "0" 23:47:58 DEBUG } 23:47:58 DEBUG ----- output ----- 23:47:58 DEBUG ] 23:47:58 DEBUG --- status code --- 23:47:58 DEBUG --- status code --- 23:47:58 DEBUG } 23:47:58 DEBUG --- http response --- 23:47:58 DEBUG ] 23:47:58 DEBUG { 23:47:58 DEBUG } 23:47:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:47:58 DEBUG } 23:47:58 DEBUG "status": "success", 23:47:58 DEBUG ] 23:47:58 DEBUG --- http response --- 23:47:58 DEBUG --- http response --- 23:47:58 DEBUG ----- output ----- 23:47:58 DEBUG ] 23:47:58 DEBUG "data": { 23:47:58 DEBUG } 23:47:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:47:58 DEBUG { 23:47:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:47:58 DEBUG } 23:47:58 DEBUG "resultType": "vector", 23:47:58 DEBUG } 23:47:58 DEBUG ----- output ----- 23:47:58 DEBUG "status": "success", 23:47:58 DEBUG ----- output ----- 23:47:58 DEBUG } 23:47:58 DEBUG "result": [ 23:47:58 DEBUG { 23:47:58 DEBUG "data": { 23:47:58 DEBUG { 23:47:58 DEBUG { 23:47:58 DEBUG "status": "success", 23:47:58 DEBUG "resultType": "vector", 23:47:58 DEBUG "status": "success", 23:47:58 DEBUG "metric": { 23:47:58 DEBUG "data": { 23:47:58 DEBUG "result": [ 23:47:58 DEBUG "data": { 23:47:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:47:58 DEBUG "resultType": "vector", 23:47:58 DEBUG { 23:47:58 DEBUG "resultType": "vector", 23:47:58 DEBUG }, 23:47:58 DEBUG "result": [ 23:47:58 DEBUG "metric": { 23:47:58 DEBUG "result": [ 23:47:58 DEBUG "value": [ 23:47:58 DEBUG { 23:47:58 DEBUG "instance": "10.142.0.107:9100" 23:47:58 DEBUG { 23:47:58 DEBUG 23:47:58 DEBUG "metric": { 23:47:58 DEBUG }, 23:47:58 DEBUG "metric": { 23:47:58 DEBUG 23:47:58 DEBUG "instance": "10.142.0.107:9100" 23:47:58 DEBUG "value": [ 23:47:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:47:58 DEBUG 23:47:58 DEBUG }, 23:47:58 DEBUG 23:47:58 DEBUG }, 23:47:58 DEBUG [...] 23:47:58 DEBUG "value": [ 23:47:58 DEBUG 23:47:58 DEBUG "value": [ 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG [...] 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:47:58 DEBUG [...] 23:47:58 DEBUG 23:47:58 DEBUG [...] 23:47:58 DEBUG }, 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG "value": [ 23:47:58 DEBUG 23:47:58 DEBUG "instance": "10.142.0.43:9100" 23:47:58 DEBUG 23:47:58 DEBUG 1713221278, 23:47:58 DEBUG 23:47:58 DEBUG }, 23:47:58 DEBUG 23:47:58 DEBUG "0" 23:47:58 DEBUG "instance": "10.142.0.43:9100" 23:47:58 DEBUG "value": [ 23:47:58 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:47:58 DEBUG ] 23:47:58 DEBUG }, 23:47:58 DEBUG 1713221278, 23:47:58 DEBUG }, 23:47:58 DEBUG } 23:47:58 DEBUG "value": [ 23:47:58 DEBUG "5600.166666666666" 23:47:58 DEBUG "value": [ 23:47:58 DEBUG ] 23:47:58 DEBUG 1713221278, 23:47:58 DEBUG ] 23:47:58 DEBUG 1713221278, 23:47:58 DEBUG } 23:47:58 DEBUG "13924066304" 23:47:58 DEBUG } 23:47:58 DEBUG "0" 23:47:58 DEBUG } 23:47:58 DEBUG ] 23:47:58 DEBUG ] 23:47:58 DEBUG ] 23:47:58 DEBUG } 23:47:58 DEBUG } 23:47:58 DEBUG } 23:47:58 DEBUG ] 23:47:58 DEBUG } 23:47:58 DEBUG ] 23:47:58 DEBUG } 23:47:58 DEBUG } 23:47:58 DEBUG } 23:47:58 DEBUG } 23:47:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713221278" 23:47:58 INFO [http_cmd]: http status code OK 23:47:58 DEBUG --- status code --- 23:47:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:58 DEBUG --- http response --- 23:47:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:47:58 DEBUG ----- output ----- 23:47:58 DEBUG { 23:47:58 DEBUG "status": "success", 23:47:58 DEBUG "data": { 23:47:58 DEBUG "resultType": "vector", 23:47:58 DEBUG "result": [ 23:47:58 DEBUG { 23:47:58 DEBUG "metric": { 23:47:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:47:58 DEBUG }, 23:47:58 DEBUG "value": [ 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG [...] 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG 23:47:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:47:58 DEBUG }, 23:47:58 DEBUG "value": [ 23:47:58 DEBUG 1713221278, 23:47:58 DEBUG "0.082" 23:47:58 DEBUG ] 23:47:58 DEBUG } 23:47:58 DEBUG ] 23:47:58 DEBUG } 23:47:58 DEBUG } 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221279" 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221279" 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221279" 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221279" 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221279" 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221279" 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221279" 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713221279" 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221279" 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221279" 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221279" 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221279" 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 DEBUG --- status code --- 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 DEBUG --- status code --- 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 DEBUG --- status code --- 23:47:59 DEBUG --- status code --- 23:47:59 DEBUG --- status code --- 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG --- status code --- 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG --- status code --- 23:47:59 DEBUG --- status code --- 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG --- http response --- 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG --- http response --- 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG --- http response --- 23:47:59 DEBUG --- http response --- 23:47:59 DEBUG --- http response --- 23:47:59 DEBUG --- status code --- 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:47:59 DEBUG --- http response --- 23:47:59 DEBUG --- http response --- 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:47:59 DEBUG --- http response --- 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:47:59 DEBUG --- http response --- 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG --- status code --- 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG { 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG { 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG { 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG { 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG { 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG { 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG { 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG { 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG --- http response --- 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG "data": { 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG "data": { 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG "data": { 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG "data": { 23:47:59 DEBUG { 23:47:59 DEBUG "data": { 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:47:59 DEBUG "resultType": "vector", 23:47:59 DEBUG "data": { 23:47:59 DEBUG "resultType": "vector", 23:47:59 DEBUG "data": { 23:47:59 DEBUG "resultType": "vector", 23:47:59 DEBUG "data": { 23:47:59 DEBUG "resultType": "vector", 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG "resultType": "vector", 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG "result": [ 23:47:59 DEBUG "resultType": "vector", 23:47:59 DEBUG "result": [ 23:47:59 DEBUG "resultType": "vector", 23:47:59 DEBUG "result": [ 23:47:59 DEBUG "resultType": "vector", 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 DEBUG "result": [ 23:47:59 DEBUG "data": { 23:47:59 DEBUG "result": [ 23:47:59 DEBUG { 23:47:59 DEBUG { 23:47:59 DEBUG "result": [ 23:47:59 DEBUG { 23:47:59 DEBUG "result": [ 23:47:59 DEBUG { 23:47:59 DEBUG "result": [ 23:47:59 DEBUG --- status code --- 23:47:59 DEBUG --- status code --- 23:47:59 DEBUG { 23:47:59 DEBUG "resultType": "vector", 23:47:59 DEBUG { 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG "metric": { 23:47:59 DEBUG { 23:47:59 DEBUG "metric": { 23:47:59 DEBUG { 23:47:59 DEBUG "metric": { 23:47:59 DEBUG { 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG "metric": { 23:47:59 DEBUG "result": [ 23:47:59 DEBUG "metric": { 23:47:59 DEBUG "data": { 23:47:59 DEBUG "pod": "ds-idrepo-2" 23:47:59 DEBUG "metric": { 23:47:59 DEBUG "pod": "ds-idrepo-2" 23:47:59 DEBUG "metric": { 23:47:59 DEBUG "pod": "ds-idrepo-2" 23:47:59 DEBUG "metric": { 23:47:59 DEBUG --- http response --- 23:47:59 DEBUG --- http response --- 23:47:59 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:47:59 DEBUG { 23:47:59 DEBUG "pod": "ds-idrepo-2" 23:47:59 DEBUG "resultType": "vector", 23:47:59 DEBUG }, 23:47:59 DEBUG "pod": "ds-idrepo-2" 23:47:59 DEBUG }, 23:47:59 DEBUG "pod": "ds-idrepo-2" 23:47:59 DEBUG }, 23:47:59 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:47:59 DEBUG }, 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:47:59 DEBUG "metric": { 23:47:59 DEBUG }, 23:47:59 DEBUG "result": [ 23:47:59 DEBUG "value": [ 23:47:59 DEBUG }, 23:47:59 DEBUG "value": [ 23:47:59 DEBUG }, 23:47:59 DEBUG "value": [ 23:47:59 DEBUG }, 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG "value": [ 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:47:59 DEBUG "value": [ 23:47:59 DEBUG { 23:47:59 DEBUG 23:47:59 DEBUG "value": [ 23:47:59 DEBUG 23:47:59 DEBUG "value": [ 23:47:59 DEBUG 23:47:59 DEBUG "value": [ 23:47:59 DEBUG { 23:47:59 DEBUG 23:47:59 DEBUG { 23:47:59 DEBUG }, 23:47:59 DEBUG 23:47:59 DEBUG "metric": { 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG "value": [ 23:47:59 DEBUG 23:47:59 DEBUG "instance": "10.142.0.107:9100" 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG "data": { 23:47:59 DEBUG 23:47:59 DEBUG "data": { 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG }, 23:47:59 DEBUG [...] 23:47:59 DEBUG 23:47:59 DEBUG [...] 23:47:59 DEBUG 23:47:59 DEBUG [...] 23:47:59 DEBUG 23:47:59 DEBUG "resultType": "vector", 23:47:59 DEBUG [...] 23:47:59 DEBUG "resultType": "vector", 23:47:59 DEBUG 23:47:59 DEBUG [...] 23:47:59 DEBUG "value": [ 23:47:59 DEBUG 23:47:59 DEBUG [...] 23:47:59 DEBUG 23:47:59 DEBUG [...] 23:47:59 DEBUG 23:47:59 DEBUG [...] 23:47:59 DEBUG "result": [ 23:47:59 DEBUG 23:47:59 DEBUG "result": [ 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG { 23:47:59 DEBUG 23:47:59 DEBUG { 23:47:59 DEBUG [...] 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG "metric": { 23:47:59 DEBUG 23:47:59 DEBUG "metric": { 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG [...] 23:47:59 DEBUG "pod": "ds-idrepo-0" 23:47:59 DEBUG "pod": "ds-idrepo-0" 23:47:59 DEBUG 23:47:59 DEBUG "pod": "ds-idrepo-0" 23:47:59 DEBUG 23:47:59 DEBUG "instance": "10.142.0.107:9100" 23:47:59 DEBUG "pod": "idm-b85f9d978-j5pbm" 23:47:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:47:59 DEBUG 23:47:59 DEBUG "pod": "ds-idrepo-0" 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG }, 23:47:59 DEBUG }, 23:47:59 DEBUG "pod": "ds-idrepo-0" 23:47:59 DEBUG }, 23:47:59 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:47:59 DEBUG }, 23:47:59 DEBUG }, 23:47:59 DEBUG }, 23:47:59 DEBUG 23:47:59 DEBUG }, 23:47:59 DEBUG "pod": "ds-idrepo-0" 23:47:59 DEBUG 23:47:59 DEBUG "value": [ 23:47:59 DEBUG "value": [ 23:47:59 DEBUG }, 23:47:59 DEBUG "value": [ 23:47:59 DEBUG }, 23:47:59 DEBUG "value": [ 23:47:59 DEBUG "value": [ 23:47:59 DEBUG "value": [ 23:47:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:47:59 DEBUG "value": [ 23:47:59 DEBUG }, 23:47:59 DEBUG 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG "value": [ 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG "value": [ 23:47:59 DEBUG 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG 23:47:59 DEBUG }, 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG "value": [ 23:47:59 DEBUG "instance": "10.142.0.43:9100" 23:47:59 DEBUG "0" 23:47:59 DEBUG "0" 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG "0" 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG 23:47:59 DEBUG "863.6163502021955" 23:47:59 DEBUG 23:47:59 DEBUG "value": [ 23:47:59 DEBUG "0" 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG }, 23:47:59 DEBUG ] 23:47:59 DEBUG ] 23:47:59 DEBUG "0" 23:47:59 DEBUG ] 23:47:59 DEBUG "1.2011108954241527" 23:47:59 DEBUG 23:47:59 DEBUG ] 23:47:59 DEBUG 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG ] 23:47:59 DEBUG "0" 23:47:59 DEBUG "value": [ 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG ] 23:47:59 DEBUG } 23:47:59 DEBUG ] 23:47:59 DEBUG [...] 23:47:59 DEBUG } 23:47:59 DEBUG [...] 23:47:59 DEBUG "0" 23:47:59 DEBUG } 23:47:59 DEBUG ] 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG ] 23:47:59 DEBUG ] 23:47:59 DEBUG } 23:47:59 DEBUG ] 23:47:59 DEBUG } 23:47:59 DEBUG 23:47:59 DEBUG ] 23:47:59 DEBUG 23:47:59 DEBUG ] 23:47:59 DEBUG ] 23:47:59 DEBUG } 23:47:59 DEBUG "12580.199999999997" 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG ] 23:47:59 DEBUG } 23:47:59 DEBUG ] 23:47:59 DEBUG 23:47:59 DEBUG } 23:47:59 DEBUG 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG ] 23:47:59 DEBUG ] 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG 23:47:59 DEBUG } 23:47:59 DEBUG 23:47:59 DEBUG ] 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG "instance": "10.142.0.43:9100" 23:47:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG ] 23:47:59 DEBUG }, 23:47:59 DEBUG }, 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG "value": [ 23:47:59 DEBUG "value": [ 23:47:59 DEBUG } 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG "0" 23:47:59 DEBUG "0" 23:47:59 DEBUG ] 23:47:59 DEBUG ] 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG ] 23:47:59 DEBUG ] 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221279" 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 DEBUG --- status code --- 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG --- http response --- 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG { 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG "data": { 23:47:59 DEBUG "resultType": "vector", 23:47:59 DEBUG "result": [ 23:47:59 DEBUG { 23:47:59 DEBUG "metric": { 23:47:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:47:59 DEBUG }, 23:47:59 DEBUG "value": [ 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG [...] 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:47:59 DEBUG }, 23:47:59 DEBUG "value": [ 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG "1234.9" 23:47:59 DEBUG ] 23:47:59 DEBUG } 23:47:59 DEBUG ] 23:47:59 DEBUG } 23:47:59 DEBUG } 23:47:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221279" 23:47:59 INFO [http_cmd]: http status code OK 23:47:59 DEBUG --- status code --- 23:47:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:47:59 DEBUG --- http response --- 23:47:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:47:59 DEBUG ----- output ----- 23:47:59 DEBUG { 23:47:59 DEBUG "status": "success", 23:47:59 DEBUG "data": { 23:47:59 DEBUG "resultType": "vector", 23:47:59 DEBUG "result": [ 23:47:59 DEBUG { 23:47:59 DEBUG "metric": { 23:47:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:47:59 DEBUG }, 23:47:59 DEBUG "value": [ 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG [...] 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG 23:47:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:47:59 DEBUG }, 23:47:59 DEBUG "value": [ 23:47:59 DEBUG 1713221279, 23:47:59 DEBUG "0" 23:47:59 DEBUG ] 23:47:59 DEBUG } 23:47:59 DEBUG ] 23:47:59 DEBUG } 23:47:59 DEBUG } 23:48:00 INFO 23:48:00 INFO [loop_until]: kubectl --namespace=xlou top pods 23:48:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:48:00 INFO [loop_until]: OK (rc = 0) 23:48:00 DEBUG --- stdout --- 23:48:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 1148m 8591Mi am-7d597cf7d6-8vr8f 1006m 8617Mi am-7d597cf7d6-9wz9h 1441m 8908Mi ds-cts-0 3644m 8571Mi ds-cts-1 2709m 8630Mi ds-cts-2 2596m 8811Mi ds-idrepo-0 1296m 5009Mi ds-idrepo-1 380m 2755Mi ds-idrepo-2 435m 2835Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1027Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 984m 760Mi 23:48:00 DEBUG --- stderr --- 23:48:00 DEBUG 23:48:01 INFO 23:48:01 INFO [loop_until]: kubectl --namespace=xlou top node 23:48:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:48:01 INFO [loop_until]: OK (rc = 0) 23:48:01 DEBUG --- stdout --- 23:48:01 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 5073Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2676m 8% 10990Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1186m 3% 10704Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 92m 0% 3110Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 542m 1% 5679Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 4001m 13% 12956Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2973Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1070m 3% 10138Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 452m 1% 4962Mi 4% gke-xlou-cdm-ds-0f601751-j488 2582m 8% 10516Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2462Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 1620m 5% 10604Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 1382m 4% 7189Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 1087m 6% 4702Mi 8% 23:48:01 DEBUG --- stderr --- 23:48:01 DEBUG 23:48:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221338" 23:48:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221338" 23:48:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221338" 23:48:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713221338" 23:48:58 INFO [http_cmd]: http status code OK 23:48:58 DEBUG --- status code --- 23:48:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:58 DEBUG --- http response --- 23:48:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:48:58 DEBUG ----- output ----- 23:48:58 DEBUG { 23:48:58 DEBUG "status": "success", 23:48:58 DEBUG "data": { 23:48:58 DEBUG "resultType": "vector", 23:48:58 DEBUG "result": [ 23:48:58 DEBUG { 23:48:58 DEBUG "metric": { 23:48:58 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:48:58 DEBUG }, 23:48:58 DEBUG "value": [ 23:48:58 DEBUG 23:48:58 DEBUG 23:48:58 DEBUG 23:48:58 DEBUG [...] 23:48:58 DEBUG 23:48:58 DEBUG 23:48:58 DEBUG 23:48:58 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:48:58 DEBUG }, 23:48:58 DEBUG "value": [ 23:48:58 DEBUG 1713221338, 23:48:58 DEBUG "0" 23:48:58 DEBUG ] 23:48:58 DEBUG } 23:48:58 DEBUG ] 23:48:58 DEBUG } 23:48:58 DEBUG } 23:48:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221338" 23:48:58 INFO [http_cmd]: http status code OK 23:48:58 DEBUG --- status code --- 23:48:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:58 DEBUG --- http response --- 23:48:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:48:58 DEBUG ----- output ----- 23:48:58 INFO [http_cmd]: http status code OK 23:48:58 DEBUG { 23:48:58 DEBUG --- status code --- 23:48:58 DEBUG "status": "success", 23:48:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:58 DEBUG "data": { 23:48:58 DEBUG --- http response --- 23:48:58 DEBUG "resultType": "vector", 23:48:58 DEBUG "result": [ 23:48:58 DEBUG { 23:48:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:48:58 DEBUG "metric": { 23:48:58 DEBUG ----- output ----- 23:48:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:48:58 DEBUG { 23:48:58 DEBUG }, 23:48:58 DEBUG "status": "success", 23:48:58 DEBUG "value": [ 23:48:58 DEBUG "data": { 23:48:58 DEBUG 23:48:58 DEBUG "resultType": "vector", 23:48:58 DEBUG 23:48:58 DEBUG "result": [ 23:48:58 DEBUG 23:48:58 DEBUG { 23:48:58 DEBUG [...] 23:48:58 DEBUG "metric": { 23:48:58 DEBUG 23:48:58 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:48:58 DEBUG 23:48:58 DEBUG }, 23:48:58 DEBUG 23:48:58 DEBUG "value": [ 23:48:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:48:58 DEBUG 23:48:58 DEBUG }, 23:48:58 DEBUG 23:48:58 DEBUG "value": [ 23:48:58 DEBUG 23:48:58 DEBUG 1713221338, 23:48:58 DEBUG [...] 23:48:58 DEBUG "1.0059339967163854" 23:48:58 DEBUG 23:48:58 DEBUG ] 23:48:58 DEBUG 23:48:58 DEBUG } 23:48:58 DEBUG 23:48:58 DEBUG ] 23:48:58 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:48:58 DEBUG } 23:48:58 DEBUG }, 23:48:58 DEBUG } 23:48:58 DEBUG "value": [ 23:48:58 DEBUG 1713221338, 23:48:58 DEBUG "13939017.602682311" 23:48:58 DEBUG ] 23:48:58 DEBUG } 23:48:58 DEBUG ] 23:48:58 DEBUG } 23:48:58 DEBUG } 23:48:58 INFO [http_cmd]: http status code OK 23:48:58 DEBUG --- status code --- 23:48:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:58 DEBUG --- http response --- 23:48:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:48:58 DEBUG ----- output ----- 23:48:58 DEBUG { 23:48:58 DEBUG "status": "success", 23:48:58 DEBUG "data": { 23:48:58 DEBUG "resultType": "vector", 23:48:58 DEBUG "result": [ 23:48:58 DEBUG { 23:48:58 DEBUG "metric": { 23:48:58 DEBUG "instance": "10.142.0.107:9100" 23:48:58 DEBUG }, 23:48:58 DEBUG "value": [ 23:48:58 DEBUG 23:48:58 DEBUG 23:48:58 DEBUG 23:48:58 DEBUG [...] 23:48:58 DEBUG 23:48:58 DEBUG 23:48:58 DEBUG 23:48:58 DEBUG "instance": "10.142.0.43:9100" 23:48:58 DEBUG }, 23:48:58 DEBUG "value": [ 23:48:58 DEBUG 1713221338, 23:48:58 DEBUG "13925114880" 23:48:58 DEBUG ] 23:48:58 DEBUG } 23:48:58 DEBUG ] 23:48:58 DEBUG } 23:48:58 DEBUG } 23:48:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221338" 23:48:58 INFO [http_cmd]: http status code OK 23:48:58 DEBUG --- status code --- 23:48:58 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:58 DEBUG --- http response --- 23:48:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713221338" 23:48:58 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:48:58 DEBUG ----- output ----- 23:48:58 DEBUG { 23:48:58 DEBUG "status": "success", 23:48:58 DEBUG "data": { 23:48:58 DEBUG "resultType": "vector", 23:48:58 DEBUG "result": [ 23:48:58 DEBUG { 23:48:58 DEBUG "metric": { 23:48:58 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:48:58 DEBUG }, 23:48:58 DEBUG "value": [ 23:48:58 DEBUG 23:48:58 DEBUG 23:48:58 DEBUG 23:48:58 DEBUG [...] 23:48:58 DEBUG 23:48:58 DEBUG 23:48:58 DEBUG 23:48:58 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:48:58 DEBUG }, 23:48:58 DEBUG "value": [ 23:48:58 DEBUG 1713221338, 23:48:58 DEBUG "1502863360" 23:48:58 DEBUG ] 23:48:58 DEBUG } 23:48:58 DEBUG ] 23:48:58 DEBUG } 23:48:58 DEBUG } 23:48:58 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221338" 23:48:58 ERROR [http_cmd]: ERROR 23:48:58 ERROR --- status code --- 23:48:58 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:48:58 ERROR --- http response --- 23:48:58 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG --- http response --- 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:48:59 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:48:59 DEBUG --- status code --- 23:48:59 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG { 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "data": { 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG [...] 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 1713221338, 23:48:59 DEBUG "0" 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221339" 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG { 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "data": { 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "pod": "ds-idrepo-2" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG [...] 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "pod": "ds-cts-2" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG "0" 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221339" 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG { 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "data": { 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG [...] 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG "19778" 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG { 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "data": { 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "instance": "10.142.0.107:9100" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG [...] 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "instance": "10.142.0.43:9100" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG "0.00040000000000001514" 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG { 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "data": { 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "instance": "10.142.0.107:9100" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG [...] 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "instance": "10.142.0.43:9100" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG "6856.9" 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG { 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "data": { 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG [...] 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG "0.082" 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713221339" 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221339" 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG { 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "data": { 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG [...] 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG "1.0348580364217632" 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG { 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "data": { 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "instance": "10.142.0.107:9100" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG [...] 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "instance": "10.142.0.43:9100" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG "0.0006666666666666673" 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG { 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "data": { 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG [...] 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "pod": "lodemon-56b95674f4-jwfm2" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG "3783.01828705129" 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713221339" 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- status code --- 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- http response --- 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG { 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG { 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG { 23:48:59 DEBUG { 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG "status": "success", 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 DEBUG "data": { 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG "data": { 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG "data": { 23:48:59 DEBUG "data": { 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG { 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "result": [ 23:48:59 DEBUG "result": [ 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:48:59 DEBUG "data": { 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:48:59 DEBUG { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "data": { 23:48:59 DEBUG { 23:48:59 DEBUG { 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "pod": "ds-idrepo-2" 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG { 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG { 23:48:59 DEBUG "pod": "ds-idrepo-2" 23:48:59 DEBUG }, 23:48:59 DEBUG "result": [ 23:48:59 DEBUG "pod": "ds-idrepo-2" 23:48:59 DEBUG "pod": "ds-idrepo-2" 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG { 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG { 23:48:59 DEBUG }, 23:48:59 DEBUG }, 23:48:59 DEBUG "data": { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "data": { 23:48:59 DEBUG "data": { 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "value": [ 23:48:59 DEBUG "value": [ 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "pod": "ds-idrepo-2" 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "result": [ 23:48:59 DEBUG }, 23:48:59 DEBUG "result": [ 23:48:59 DEBUG "result": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG }, 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG { 23:48:59 DEBUG "value": [ 23:48:59 DEBUG { 23:48:59 DEBUG { 23:48:59 DEBUG 23:48:59 DEBUG [...] 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "metric": { 23:48:59 DEBUG 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG [...] 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG [...] 23:48:59 DEBUG [...] 23:48:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:48:59 DEBUG 23:48:59 DEBUG "pod": "ds-idrepo-2" 23:48:59 DEBUG "instance": "10.142.0.107:9100" 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG }, 23:48:59 DEBUG 23:48:59 DEBUG }, 23:48:59 DEBUG }, 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "value": [ 23:48:59 DEBUG [...] 23:48:59 DEBUG "value": [ 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG "pod": "ds-idrepo-0" 23:48:59 DEBUG [...] 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "pod": "ds-idrepo-0" 23:48:59 DEBUG }, 23:48:59 DEBUG 23:48:59 DEBUG "pod": "ds-idrepo-0" 23:48:59 DEBUG "pod": "ds-idrepo-0" 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG }, 23:48:59 DEBUG }, 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG 23:48:59 DEBUG "value": [ 23:48:59 DEBUG "value": [ 23:48:59 DEBUG [...] 23:48:59 DEBUG "pod": "ds-idrepo-0" 23:48:59 DEBUG [...] 23:48:59 DEBUG [...] 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG "0" 23:48:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG 23:48:59 DEBUG }, 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "0" 23:48:59 DEBUG ] 23:48:59 DEBUG }, 23:48:59 DEBUG "0" 23:48:59 DEBUG "0" 23:48:59 DEBUG 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG "value": [ 23:48:59 DEBUG ] 23:48:59 DEBUG ] 23:48:59 DEBUG 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG 1713221339, 23:48:59 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713221339" 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:48:59 DEBUG "0" 23:48:59 DEBUG "pod": "ds-idrepo-0" 23:48:59 DEBUG "instance": "10.142.0.43:9100" 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG "0" 23:48:59 DEBUG ] 23:48:59 DEBUG ] 23:48:59 DEBUG }, 23:48:59 DEBUG ] 23:48:59 DEBUG }, 23:48:59 DEBUG }, 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 DEBUG "value": [ 23:48:59 DEBUG } 23:48:59 DEBUG "value": [ 23:48:59 DEBUG "value": [ 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG ] 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG ] 23:48:59 DEBUG "0" 23:48:59 DEBUG } 23:48:59 DEBUG "0" 23:48:59 DEBUG "12485.866666666665" 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG } 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG } 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG { 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "data": { 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG [...] 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG "0" 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:48:59 INFO [http_cmd]: http status code OK 23:48:59 DEBUG --- status code --- 23:48:59 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:48:59 DEBUG --- http response --- 23:48:59 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:48:59 DEBUG ----- output ----- 23:48:59 DEBUG { 23:48:59 DEBUG "status": "success", 23:48:59 DEBUG "data": { 23:48:59 DEBUG "resultType": "vector", 23:48:59 DEBUG "result": [ 23:48:59 DEBUG { 23:48:59 DEBUG "metric": { 23:48:59 DEBUG "pod": "am-7d597cf7d6-8ctft" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG [...] 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG 23:48:59 DEBUG "pod": "am-7d597cf7d6-9wz9h" 23:48:59 DEBUG }, 23:48:59 DEBUG "value": [ 23:48:59 DEBUG 1713221339, 23:48:59 DEBUG "1196.3333333333333" 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG ] 23:48:59 DEBUG } 23:48:59 DEBUG } 23:49:00 INFO 23:49:00 INFO [loop_until]: kubectl --namespace=xlou top pods 23:49:00 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:49:00 INFO [loop_until]: OK (rc = 0) 23:49:00 DEBUG --- stdout --- 23:49:00 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 947m 8586Mi am-7d597cf7d6-8vr8f 905m 8614Mi am-7d597cf7d6-9wz9h 1099m 8905Mi ds-cts-0 3816m 8547Mi ds-cts-1 2224m 8632Mi ds-cts-2 2429m 8815Mi ds-idrepo-0 653m 5009Mi ds-idrepo-1 49m 2755Mi ds-idrepo-2 59m 2834Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1027Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-56b95674f4-jwfm2 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1015m 1564Mi 23:49:00 DEBUG --- stderr --- 23:49:00 DEBUG 23:49:01 INFO 23:49:01 INFO [loop_until]: kubectl --namespace=xlou top node 23:49:01 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:49:01 INFO [loop_until]: OK (rc = 0) 23:49:01 DEBUG --- stdout --- 23:49:01 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5068Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2281m 7% 10999Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1060m 3% 10700Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3090Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 117m 0% 5697Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 3636m 12% 12951Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2977Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 929m 3% 10137Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 114m 0% 5004Mi 4% gke-xlou-cdm-ds-0f601751-j488 2332m 7% 10515Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2459Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 1209m 4% 10593Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 435m 1% 7200Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 1082m 6% 5500Mi 9% 23:49:01 DEBUG --- stderr --- 23:49:01 DEBUG 23:49:49 INFO Finished: True 23:49:49 INFO Waiting for threads to register finish flag 23:50:01 INFO Done. Have a nice day! :) 127.0.0.1 - - [15/Apr/2024 23:50:01] "GET /monitoring/stop HTTP/1.1" 200 - 23:50:04 WARNING File /tmp/lodemon_data-20240415_230746/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6521', ...) 127.0.0.1 - - [15/Apr/2024 23:50:08] "GET /monitoring/process HTTP/1.1" 200 -